Vehicle display device, display control method, and rearview monitoring system

ABSTRACT

Vehicle display device ( 10 ) including a display unit ( 50 ), installed in a cabin of a vehicle and having a display face oriented in a different direction from a direction of a driver (DR), configured to display an image depicting a surrounding area of the vehicle, and a mirror unit ( 55 ) installed in the cabin of the vehicle and configured to reflect some or all of an image area displayed on the display unit ( 50 ). A visible range of the surrounding area visible to the driver through an image of the display unit ( 50 ) reflected in the mirror unit ( 55 ) changes according to movement of a viewing position of the driver (DR) with respect to the mirror unit ( 55 ).

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Japanese Priority PatentApplication JP 2014-164179 filed Aug. 12, 2014, the entire contents ofwhich are incorporated herein by reference.

TECHNICAL FIELD

The present technology relates to a vehicle display device, a displaycontrol method, and a rearview monitoring system, and enables easyviewing of the area around a vehicle.

BACKGROUND ART

Recently, functions corresponding to the rearview mirror of a vehicleare being realized by a monitoring system made up of an image capturedevice and a display device. With a viewing method using a rearviewmirror, a driver is able to move the range that may be viewed(hereinafter designated the “visible range”) by moving his or her head,eye position, or the like (hereinafter simply designated the “headposition”). However, with a monitoring system that simply displays animage acquired with an image capture device on a display device, thevisible range is fixed, and the driver is unable to move the range thatmay be viewed, even if the driver moves his or her head position. Forthis reason, Patent Literature 1 discloses the moving of the visiblerange according to a change in head position by detecting a change inthe head position of the driver, trimming an image from a captured imageof the surrounding area according to the change in the head position,and displaying the trimmed image on a display device.

CITATION LIST Patent Literature PTL 1: JP 2010-179850A SUMMARY TechnicalProblem

Meanwhile, when trimming an image from an image of the surrounding areaaccording to a change in the head position, to alter the visible rangeaccording to the change in head position similarly to a rearview mirror,the image should be trimmed appropriately according to the change in thehead position. For example, if there is little movement of the imagetrimming position with respect to a change in the head position, thedesired range may not be displayed even after moving one's headsimilarly to the case of using a rearview mirror to view the desiredrange.

Accordingly, it is desirable to provide a vehicle display device,display control method, and rearview monitoring system that enables easyviewing of the area around a vehicle.

Solution to Problem

An embodiment of the present technology is a vehicle display deviceincluding a display unit, installed in a cabin of a vehicle and having adisplay face oriented in a different direction from a direction of adriver, configured to display an image depicting a surrounding area ofthe vehicle, and a mirror unit installed in the cabin of the vehicle andconfigured to reflect some or all of an image area displayed on thedisplay unit. A visible range of the surrounding area visible to thedriver through an image of the display unit reflected in the mirror unitchanges according to movement of a viewing position of the driver withrespect to the mirror unit.

In the present technology, a display unit and a mirror unit are providedin the cabin of a vehicle. The display face of the display unit isoriented in a different direction from the direction of the driver, andsome or all of the image area of the display face is visible to thedriver via the mirror unit. On the display unit, an image depicting thearea around the vehicle is displayed. Also, of the surrounding area, thevisible range that may be viewed by the image of the display unitreflected by the mirror unit changes according to movement of theviewing position of the driver. For example, the brightness is loweredfor other display areas except for a display area corresponding to thevisible range. Also, movement of the driver is determined, and on thebasis of the movement determination result, the display area to belowered in brightness is controlled to move or extend the visible rangeaccording to the movement of the driver. Also, image compression isconducted in the movement direction of the visible range with respect toan edge area in the movement direction of the visible range in an imagedepicting the area around a vehicle, and on the basis of a movementdetermination result, the compression ratio or the compression range iscontrolled to move or extend the visible range. Also, the display of animage depicting a surrounding area is conducting in response to thedriver glancing at the mirror unit on the basis of a movementdetermination result, for example, and the area of the visible range ismoved or extended in the case of determining that the driver performed apredefined movement after the detection of the glance. Furthermore, thevisible range is extended according to the driver's movement, and thearea size of the visible range to extend is controlled on the basis ofthe driving conditions of the vehicle. A predefined movement by thedriver refers to gesture control that detects instructions while alsoaccounting for the driver's gesture features, and an exemplaryembodiment will be discussed in detail later.

Another embodiment of the present technology is a display control methodincluding displaying an image depicting a surrounding area of a vehicleon a display unit installed in a cabin of the vehicle and having adisplay face oriented in a different direction from a direction of adriver, and with a display controller, determining motion of the driverand controlling a display of an image to display on the display unit onthe basis of a motion determination result, in a manner that a visiblerange visible to the driver though an image on the display unitreflected in a mirror unit provided in the cabin of the vehicle changesaccording to the motion of the driver.

Another embodiment of the present technology is a rearview monitoringsystem including at least two or more image capture devices attached toa vehicle exterior and facing rearward with respect to a vehicle traveldirection, and a display unit provided with at least two or moreadjacently arranged screen display areas in a dashboard center in acabin of the vehicle. In displayed content of captured images to displayin the screen display areas, an image from a first image capture deviceinstalled on a front side in the vehicle travel direction is arrangedabove an image from an image capture device installed farther rearwardin the vehicle travel direction than the first image capture device.

Another embodiment of the present technology is a rearview monitoringsystem including at least two or more image capture devices attached toa vehicle exterior and facing rearward with respect to a vehicle traveldirection, and a display unit provided with at least two or moreadjacently arranged screen display areas in a cabin of the vehicle. Indisplayed content of captured images to display in the screen displayareas, a vanishing point at infinity included in an image from a firstimage capture device installed on a front side in the vehicle traveldirection is arranged above a vanishing point at infinity included in animage from an image capture device installed farther rearward in thevehicle travel direction than the first image capture device.

Advantageous Effects of Invention

According to one or more embodiments of the present technology, an imagedepicting the area around a vehicle is displayed on a display unit thatis installed in the cabin of the vehicle and whose display face isoriented in a different direction from the direction of the driver. Amirror unit is installed in the cabin of the vehicle and reflects someor all of the image area displayed on the display unit. The visiblerange of the surrounding area that is visible to the driver through theimage of the display unit reflected in the mirror unit changes accordingto movement of the viewing position of the driver with respect to themirror unit. For this reason, when an image of the area around a vehicleis captured and displayed on the display unit so that the surroundingarea may be checked, a desired range of the surrounding area, such as aview of the rear and surroundings, for example, may be checked easily ina similar sense as a optical rearview mirror of the past. Note that theadvantageous effects described in this specification are merely for thesake of example and non-limiting, and there may be additionaladvantageous effects.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an example of a configuration of afirst embodiment.

FIG. 2 is a diagram illustrating an example of a positional relationshipbetween a driver, a display unit, and a mirror unit.

FIG. 3 is a diagram for explaining an image capture range of asurrounding area image capture unit.

FIG. 4 is a diagram illustrating a relationship between a mirror unitand display unit, and a rearview mirror of the past.

FIG. 5 is a diagram typifying the relationship between change of focusand reaction time.

FIG. 6 is a diagram for explaining operation of a first embodiment.

FIG. 7 is a diagram illustrating an example of a case in which themirror unit includes a curved part.

FIG. 8 is a diagram illustrating an example of a configuration of asecond embodiment.

FIG. 9 is a diagram illustrating an example of a positional relationshipbetween a driver, a display unit, and a mirror unit.

FIG. 10 is a diagram illustrating a configuration of a displaycontroller.

FIG. 11 is a diagram for explaining operation of a second embodiment.

FIG. 12 is a flowchart illustrating operation of a display controller.

FIG. 13 is a diagram for explaining a first display control operation ofa display controller.

FIG. 14 is a diagram for explaining a second display control operationof a display controller.

FIG. 15 illustrates a relationship between the display and compressionratio of a display unit, and the surrounding area.

FIG. 16 is a diagram illustrating an example of a configuration of athird embodiment.

FIG. 17 is a diagram illustrating an example of the arrangement ofsurrounding area image capture units.

FIG. 18 is a diagram illustrating a configuration of a displaycontroller.

FIG. 19 is a flowchart illustrating operation of a display controller.

FIG. 20 is a diagram illustrating an example of visible range modes.

FIG. 21 is a diagram illustrating an example of a relationship betweendriving conditions determination results and visible range modes.

FIG. 22 is a diagram illustrating an example of a display imagedisplayed on a display unit.

FIG. 23 is a diagram illustrating another example of a display imagedisplayed on a display unit.

FIG. 24 is a diagram illustrating another example of a display imagedisplayed on a display unit.

FIG. 25 is a flowchart illustrating an example of operation in a case inwhich the driver grasps the conditions of the surrounding area.

FIG. 26 is a diagram illustrating an example of a configuration of afourth embodiment.

FIG. 27 is a diagram illustrating a configuration of a displaycontroller.

FIG. 28 is a flowchart illustrating operation of a display controller.

FIG. 29 is a diagram illustrating an example of the operation of aglance and an instruction to change the visible range.

FIG. 30 is a diagram illustrating an example of the operation of glancedetermination and determination of an instruction to change the visiblerange.

FIG. 31 is a diagram illustrating an example of a case of using theacceleration of a head rotation operation as an operating instructionmagnitude.

FIG. 32 illustrates an example of a configuration of a displaycontroller in the case of presenting a warning display.

FIG. 33 is a diagram illustrating an example of the placement of asurrounding area image capture unit and a surrounding image displayed ona display unit.

FIG. 34 is a diagram for explaining the changing of a surrounding areaimage when the head moves in a front-back direction.

FIG. 35 is a diagram for explaining the changing of a surrounding areaimage when the head moves in a leftward direction.

FIG. 36 is a diagram illustrating an example of a configuration in thecase of using a single surrounding area image capture unit to realize afunction corresponding to an inner rearview mirror.

FIG. 37 is a diagram illustrating an example of a configuration in thecase of using multiple surrounding area image capture units to realize afunction corresponding to an inner rearview mirror.

FIG. 38 is a diagram illustrating an example of a configuration in thecase of reducing the blind spot compared to an inner rearview mirror.

FIG. 39 is a diagram for explaining the changing of an image joiningposition.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present technology will be described.Note that the description will proceed in the following order.

1. First embodiment

1-1. Configuration of first embodiment

1-2. Operation of first embodiment

1-3. Other configurations and operation of first embodiment

2. Second embodiment

2-1. Configuration of second embodiment

2-2. Operation of second embodiment

3. Third embodiment

3-1. Configuration of third embodiment

3-2. Operation of third embodiment

4. Fourth embodiment

4-1. Configuration of fourth embodiment

4-2. Operation of fourth embodiment

5. Other embodiments

1. First Embodiment 1-1. Configuration of First Embodiment

FIG. 1 illustrates an example of a configuration of the firstembodiment. A display device 10 is equipped with a surrounding areaimage capture unit 11, a display unit 50, and a mirror unit 55. Also,FIG. 2 is a diagram illustrating an example of a positional relationshipbetween a driver, a display unit, and a mirror unit.

The surrounding area image capture unit 11 captures an image of thesurrounding area around a vehicle, and outputs an image signal to thedisplay unit 50. FIG. 3 is a diagram for explaining an image capturerange of a surrounding area image capture unit. For example, supposethat the range of the area ARb1 among the surrounding area may be viewedby the driver DR at the head position (the position of the eyes) PS1when using a rearview mirror 91. Note that in the following description,the torso position and the position of the eyes will be simply referredto as the head position.

Suppose that the range of the area ARb2 among the surrounding area maybe viewed when the driver DR moves the head position to a position PS2.The surrounding area image capture unit 11 captures an image of therange of an area ARc that includes the area ARb1 and the area ARb2, forexample, and generates an image signal. In this way, by configuring theimage capture range, the surrounding area image capture unit 11 is ableto generate an image signal depicting the surrounding area that isvisible when using the rearview mirror 91. Note that the image capturerange of the surrounding area image capture unit 11 is not limited tothe range of the area ARc that includes the area ARb1 and the area ARb2,and may be taken to be the range with a wider angle of view than thearea ARc, for example. Hereinafter, the area captured by the surroundingarea image capture unit 11 will be designated the captured surroundingarea.

The display unit 50 is installed so that the display face is indirectlyvisible to the driver DR via the mirror unit 55, and displays on thedisplay face an image captured by the surrounding area image captureunit 11 (hereinafter designated the “captured image of the surroundingarea”). Note that in a display image of the display unit 50, the imagearea that corresponds to the range of the surrounding area that isvisible to the driver via the mirror unit 55 (hereinafter designated the“visible range”) will be referred to as the monitor image area.

The mirror unit 55 is provided so that the display face of the displayunit 50 is indirectly visible to the driver DR. The mirror unit 55 isplaced inside the vehicle, at a position near a pillar, for example, sothat an image reflected in the mirror unit 55 is visible when the driverDR performs an operation similar to the operation of looking at arearview mirror of the past, for example.

In addition, the display unit 50 with respect to the mirror unit 55 isprovided with a display size and a spacing with the mirror unit 55 sothat a captured image of the surrounding area on the display unit 50reflected in the mirror unit 55 is similar to the surrounding areareflected in a rearview mirror of the past. Furthermore, the displayunit 50 and the mirror unit 55 are arranged so that the visible range ofthe surrounding area that is visible to the driver from the image of thedisplay unit 50 reflected in the mirror unit 55 is changed according tomovement of the viewing position of the driver with respect to themirror unit 55.

FIG. 4 illustrates a relationship between a mirror unit and displayunit, and a rearview mirror of the past. The mirror plane size andposition of the mirror unit 55 are provided to approximately match themirror plane range (the range of the arrow WA) of a rearview mirror 91when the driver DR looks at the rearview mirror 91. In addition, in theimage area of the display unit 50 visible via the mirror unit 55, acaptured image of the surrounding area reflected in the mirror plane ofthe rearview mirror 91 is displayed. According to such a configuration,the display unit 50 and the mirror unit 55 may be used to obtainoperational effects that are similar to a rearview mirror 91 of thepast. Although the practical example herein cites an example of a side(rearview) mirror positioned near the A-pillar of the past and thepositional relationship of the mirror unit 55 to be arranged to achievea similar visibility direction, an arrangement in which the visibilitydirection comes to the dashboard center, similarly to the visibilitydirection of a fender mirror, is also acceptable, and as an arrangementthat realizes fewer neck-turning operations by the driver, the advantageof an effective burden reduction is also anticipated.

For the display device 10, in order for a captured image of thesurrounding area to be indirectly visible to the driver DR via themirror unit 55, the distance from the driver DR to the display face ofthe display unit 50 is increased compared to the case of the driver DRdirectly viewing the display face on the display unit 50.

FIG. 5 is a diagram typifying the relationship between a change of focusin the eyes of the driver, and reaction time. As illustrated in FIG.5(A), when switching the focus of the eyes between the two points of avisible object OBa at infinity and a visible object OBb at close range,the reaction time to adjust focus from one object to the other variesdepending on the distance LA to the visible object OBb at close range.FIG. 5(B) illustrates the relationship between the distance LA to thevisible object OBb at close range and the reaction time TR. After thedistance to the visible object OBb at close range becomes shorter than acertain distance, there is a tendency for the reaction time TR to becomelonger as the distance to the visible object OBb becomes shorter.Herein, the solid line is for the case of the elderly, while the dashedline is for the middle-aged, and the one-dot chain line is for theyoung. For this reason, in the present technology, by increasing thedistance from the driver DR to the display face of the display unit 50in order for a captured image of the surrounding area to be indirectlyvisible to the driver DR via the mirror unit 55, fast perception workbecomes possible. Specifically, the display unit 50 and the mirror unit55 are arranged or optically designed so that the optical viewingdistance from the driver DR to the display face of the display unit 50via the mirror unit 55 becomes at least 1.1 m or greater. By arrangingthe display unit 50 and the mirror unit 55 in this way, with the displaydevice 10, the time over which the driver adjusts focus to a capturedimage of the surrounding area displayed on the display unit 50 isbrought closer to the time over which the driver adjusts focus to anobject seen via a rearview mirror.

In addition, in the display device 10, since a captured image of thesurrounding area is indirectly visible to the driver DR via the mirrorunit 55, the display unit 50 is arranged so that the display face andilluminating light from the display face are not directly visible to thedriver DR. In addition, a configuration provided with a shield thatblocks the display face of the display unit 50 and illuminating lightfrom the driver DR may also be used.

1-2. Operation of First Embodiment

FIG. 6 is a diagram for explaining operation of a first embodiment. Thedisplay unit 50 displays a captured image that has been captured by thesurrounding area image capture unit 11. The mirror unit 55 is configuredas a planar mirror, for example, and sized so that a partial area GRm ofthe display area GRc of the display unit 50 is visible when the driverDR indirectly views a captured image of the surrounding area via themirror unit 55. Note that the area GRm is designated the monitor imagearea. Also, the display unit 50 displays an image of the area ARccaptured by the surrounding area image capture unit 11 in the displayarea GRc.

The display unit 50 and the mirror unit 55 are arranged so that when thedriver DR of the vehicle moves his or her head position to change thevisible range, such as by moving in the direction of the arrow Va, forexample, the monitor image area GRm that is visible via the mirror unit55 moves in the direction of the arrow Vb, similarly to the case of arearview mirror of the past. Specifically, the display unit 50 and themirror unit 55 are arranged so that when the head position is moved, themovement direction of the monitor image area GRm becomes the lengthwisedirection of the display area on the display unit 50.

At this point, when the head position of the driver DR facing in thedirection of the mirror unit 55 is at the position illustrated in thedrawing, the driver DR is able to check the conditions of the visiblerange ARm of the surrounding area by viewing the image of the monitorimage area GRm via the mirror unit 55.

Next, if the driver DR moves his or her head position in the directionof the arrow Vaf pointing towards the front of the vehicle in order tocheck an area farther outward than the visible range ARm, the monitorimage area GRm visible to the driver DR via the mirror unit 55 moves inthe direction of the arrow Vbf. In other words, the driver DR, by movinghis or her head position towards the front of the vehicle and viewingthe image in the monitor image area via the mirror unit 55, is able tocheck the conditions of an area farther outward in the surrounding areathan before moving the head position.

Additionally, if the driver DR moves his or her head position towardsthe rear of the vehicle (the opposite direction of the arrow Vat), themonitor image area GRm visible to the driver DR via the mirror unit 55moves in the opposite direction of the arrow Vbf. In other words, thedriver DR, by moving his or her head position towards the rear of thevehicle and viewing the image in the monitor image area via the mirrorunit 55, is able to check the conditions of an area farther inward inthe surrounding area than before moving the head position.

Consequently, by causing the driver to view an image of the display unit50 via the mirror unit 55, a desired area of the surrounding area may bechecked similarly to the case of using a rearview mirror. In addition,since the display unit 50 and the mirror unit 55 are provided in thecabin of the vehicle, the surrounding area may be checked well, withoutbeing adversely affected by the side windows as in the case of using arearview mirror. For example, it is possible to avoid difficulty inchecking the surrounding area due to fogging or the adherence ofraindrops or the like on the side windows, for example.

1-3. Other Configurations and Operation of the First Embodiment

Meanwhile, although the mirror unit 55 according to the first embodimentdiscussed above is described as being configured as a planar mirror, themirror unit 55 is not limited to being a planar mirror, and may also beconfigured to include a curved part. By partially curving the mirrorunit 55, an advantageous effect corresponding to a convex rearviewmirror may be obtained.

FIG. 7 illustrates an example of a case in which the mirror unitincludes a curved part. The shape of the mirror unit 55 in the movementdirection of the visible range, or in other words, in the movementdirection of the monitor image area GRm according to movement of thehead position of the driver DR, is made to be a curved shape projectingout in the direction of the driver DR, for example. In addition, in themirror unit 55, the curvature of a central part MRc is less than at theends of the movement direction of the visible range, and made to be anapproximately planar shape.

By curving the ends in this way, the image becomes compressed in themovement direction of the visible range at the curved part.Consequently, if the central part MRc of the mirror unit 55 is taken tobe the area corresponding to the monitor image area GRm, the part of thesurrounding area that is farther outward or inward from the visiblerange ARm corresponding to the monitor image area GRm may be checkedwith the compressed image at the ends of the mirror unit 55.Consequently, it is possible to ensure a wider area of the surroundingarea that may be checked by the driver, compared to the case of using aplanar mirror.

2. Second Embodiment 2-1. Configuration of Second Embodiment

FIG. 8 illustrates an example of a configuration of a second embodiment.A display device 10 is equipped with a surrounding area image captureunit 11, a driver image capture unit 12, a display controller 20, adisplay unit 50, and a mirror unit 55. Meanwhile, FIG. 9 is a diagramillustrating an example of a positional relationship between a driver, adisplay unit, a mirror unit, and a driver image capture unit.

The surrounding area image capture unit 11 captures an image of thesurrounding area around a vehicle, and outputs an image signal to thedisplay controller 20. Note that the area captured by the surroundingarea image capture unit 11 will be designated the captured surroundingarea.

The driver image capture unit 12 is provided in front of the driver DRor in the direction in which the mirror unit 55 is installed, forexample, to enable determination of properties such as the headposition, head orientation (corresponding to the orientation of theface), and line of sight orientation of the driver DR. The driver imagecapture unit 12 captures an image of the driver DR, and outputs an imagesignal to the display controller 20.

The display controller 20 causes the display unit 50 to display acaptured image of the surrounding area that has been captured by thesurrounding area image capture unit 11. In addition, the displaycontroller 20 controls the display of an image displayed on the displayunit 50 according to predetermined display change communicationoperations, on the basis of properties such as the head position, headorientation, and line of sight orientation of the driver, as well asmovement of the position and orientation.

The display unit 50 is arranged so that the display face of the displayunit 50 is indirectly visible to the driver via the mirror unit 55.Also, the display face of the display unit 50 is configured to a size sothat a display image of the display unit 50 is visible via the mirrorunit 55, even if the driver DR moves his or her head position to checkan area over a wide range with a rearview mirror while driving, forexample. Note that in a display image of the display unit 50, the areathat corresponds to the visible range of the surrounding area that thedriver checks via the mirror unit 55 will be referred to as the monitorimage area.

The mirror unit 55 is provided so that the display face of the displayunit 50 is indirectly visible to the driver DR. The mirror unit 55 isplaced inside the vehicle, at a position near a pillar, for example, sothat an image reflected in the mirror unit 55 is visible when the driverDR performs an operation of looking at a rearview mirror of the past,for example. In addition, the positional relationship of the mirror unit55 with the display unit 50 and the size of the mirror unit 55 areconfigured so that when the driver DR indirectly views a captured imageof the surrounding area via the mirror unit 55, the entire display areaof the display unit 50 is reflected in the mirror unit 55. Furthermore,in the display unit 50 and the mirror unit 55, the visible range of thesurrounding area that is visible to the driver from the image of thedisplay unit 50 reflected in the mirror unit 55 is changed by displaycontrol of the display controller 20, according to movement of theviewing position of the driver with respect to the mirror unit 55. Asillustrated in FIG. 4, the mirror plane size and position of the mirrorunit 55 are provided to approximately match the mirror plane range (therange of the arrow WA) of a rearview mirror 91 when the driver DR looksat the rearview mirror 91. According to such a configuration, thedisplay unit 50 and the mirror unit 55 may be used to obtain operationaleffects that are similar to a rearview mirror 91 of the past.

For the display device 10, in order for a captured image of thesurrounding area to be indirectly visible to the driver DR via themirror unit 55, the distance from the driver DR to the display face ofthe display unit 50 is increased compared to the case of the driver DRdirectly viewing the display face on the display unit 50.

In addition, in the display device 10, since a captured image of thesurrounding area is indirectly visible to the driver DR via the mirrorunit 55, the display unit 50 is arranged so that the display face andilluminating light from the display face are not visible to the driverDR. In addition, a configuration provided with a shield that blocksdirect view of the display face of the display unit 50 and illuminatinglight from the driver DR may also be used. Although the practicalexample herein describes the positional relationship of the mirror unit55 to be arranged to achieve a visibility direction similar to arearview mirror positioned near the A-pillar of the past, an arrangementin which the line of sight comes to the dashboard center with avisibility direction corresponding to a fender mirror is alsoacceptable.

FIG. 10 is a diagram illustrating a configuration of a displaycontroller. The display controller 20 is equipped with a driver movementdetermination unit 21, a control processor 35, a display adjuster 41,and a brightness adjuster 42.

The driver movement determination unit 21, on the basis of an imagesignal supplied from the driver image capture unit 12, detects the headposition of the driver, and determines the movement direction andmovement magnitude of the head position of the driver. For example, thedriver movement determination unit 21 conducts facial recognition of thedriver on the basis of an image signal supplied from the driver imagecapture unit 12, for example, and determines the position of therecognized face, the orientation of the face (corresponding to theorientation of the head), and the orientation of the line of sight.Furthermore, the driver movement determination unit 21 tracks therecognized face, and determines the movement direction and movementmagnitude of the head position. The driver movement determination unit21 outputs a determination result to the control processor 35.

The control processor 35, on the basis of a determination result fromthe driver movement determination unit 21, generates a control signal tocontrol the display differently for the monitor image area and the otherarea (hereinafter called the “non-monitor image area”) of the displayunit 50, and outputs to the display adjuster 41 and the brightnessadjuster 42.

The display adjuster 41, on the basis of a control signal from thecontrol processor 35, conducts a scale adjustment of the captured imageof the surrounding area on an image signal supplied from the surroundingarea image capture unit 11, such as by applying image compression to thenon-monitor image area, for example.

The brightness adjuster 42, on the basis of a control signal from thecontrol processor 35, lowers the brightness of the non-monitor imagearea on the display unit 50 to less than the monitor image area. If thedisplay unit 50 is configured using display elements that may requireillumination, such as liquid crystal elements, for example, thebrightness adjuster 42 controls a partial area of the illumination, suchas the backlight of a liquid crystal panel, for example, to lower thebrightness of the non-monitor image area to less than the monitor imagearea. Meanwhile, if the display unit 50 is configured using displayelements that do not require illumination or using light-emittingelements, such as organic EL display elements, for example, a processmay be conducted to lower the signal level of a luma signalcorresponding to the non-monitor image area.

2-2. Operation of Second Embodiment

FIG. 11 is a diagram for explaining operation of a second embodiment.The display controller 20 displays an image of the area ARc captured bythe surrounding area image capture unit 11 in the display area GRc ofthe display unit 50. The mirror unit 55 is configured as a planarmirror, for example, and sized so that the display area GRc of thedisplay unit 50 is visible when the driver DR indirectly views acaptured image of the surrounding area via the mirror unit 55.

In addition, the display controller 20 conducts a brightness control,compression process, and the like on the display image in the displayarea GRc, and presents a display in the display area GRc thatdistinguishably displays an image of the visible range, which is therange of the surrounding area visible to the driver via the mirror unit55. Herein, the display area corresponding to the visible range isdesignated the monitor image area GRm.

Furthermore, the display controller 20 moves the monitor image area GRmin the direction of the arrow Vb, on the basis of the movement of thedriver DR captured by the driver image capture unit 12, such as movementin the front-back direction of the vehicle (the direction of the arrowVa), for example. Alternatively, instead of moving the monitor imagearea GRm, the display controller 20 moves the visible range ARmcorresponding to the monitor image area GRm, extends the monitor imagearea GRm in the direction of the arrow Vb, or the like.

Regarding the movement speed of an image, if a sudden change occurs inthe displayed content of the image while the driver is momentarilylooking at something other than the monitor image, for example, there isa risk of losing one's grasp of the situation. Accordingly, suddenscreen movement is avoided, and the average movement of a display imageis kept within a maximum of 0.2 s per total movement magnitude (themagnitude of a shift in the entire angle of view). Herein, by stayingwithin a screen change reaction delay time of 200 ms associated with theposition detection of the head and line of sight discussed later, it ispossible to avoid or moderate a reduced sense of user control comparedto the case of using a rearview mirror of the related art, while at thesame time, it is possible to satisfactorily avoid the overlooking ofdangers due to a delay in grasping the surrounding conditions.

It is also anticipated that the driver will perform an operation toobtained direct visual information through a window that differs fromthe information about the rear surroundings during the slight intervalafter giving an instruction to change the screen display area. In thiscase, the driver looks in the window direction past the mirror unit 55,and after momentarily taking one's attention off the screen, once againreturns to the step of checking the display screen via the mirror unit55. Consequently, if the entire screen content has changed during thisinterval, there is a risk of losing one's grasp of the situation.Accordingly, the display range due to such screen changes keeps at least¼ of the originally display content after a change, and keeps tomovement in which part of the screen from before the change is includedon the screen after the change. For example, when the driver transfersattention to other visual information (for example, the direct visualfield seen past the mirror unit 55 and the window) while the screen ischanging and moving, and then returns his or her line of sight to thedisplay unit 50, the continuity of the remembered screen may reduce theloss of one's grasp on the display unit that may occur momentarily.

FIG. 12 is a flowchart illustrating operation of a display controller.In step ST1, the display controller 20 determines whether or not anoperation of checking the surrounding area has been performed. Herein,an operation of checking the surrounding area refers to the overalloperation of the driver viewing a screen on a display device via amonitoring system made up of an image capture device and a displaydevice. The display controller 20, on the basis of an image signalsupplied from the driver image capture unit 12, determines whether ornot the driver's head orientation and line of sight direction are in thedirection of the mirror unit 55, for example. If the driver's headorientation and line of sight direction are in the direction of themirror unit 55, such as when the driver glances in the direction of themirror unit 55, for example, the display controller 20 determines thatan operation of checking the surrounding area has been performed, andproceeds to step ST2. Meanwhile, if the driver's head orientation andline of sight direction are not in the direction of the mirror unit 55,the display controller 20 determines that an operation of checking thesurrounding area has not been performed, and returns to step ST1.

In step ST2, the display controller 20 determines the visible range. Thedisplay controller 20, on the basis of an image signal supplied from thedriver image capture unit 12, detects the driver's head position, forexample, determines the visible range that is visible from the detectedhead position via the mirror unit 55, and proceeds to step ST3.

In step ST3, the display controller 20 conducts an image display controlprocess. The display controller 20 displays an image of the visiblerange on the display unit 50 as an image in the monitor image area. Inaddition, if a high-brightness subject is depicted in the non-monitorimage area, the display controller 20 controls the partial backlight orthe brightness level, for example, so that the driver's vision is notimpaired by the display content of the high level of brightness emittedfrom the image. In addition, when treating a predetermined area of thedisplay unit 50 as a monitor image area, the display controller 20conducts a process such as compressing the image of the surrounding areacorresponding to the non-monitor image area, so that the desired visiblerange may be checked from the image in the monitor image area. Thedisplay controller 20 conducts an image display control process, andproceeds to step ST4.

In step ST4, the display controller 20 determines whether or not aninstruction to change the visible range has been given. The displaycontroller 20, on the basis of an image signal supplied from the driverimage capture unit 12, determines the driver's head position, forexample, and determines whether or not an instruction operation has beenperformed to change the visible range of the surrounding area. If thedriver's head position produced a predetermined motion, the displaycontroller 20 determines that an instruction to change the visible rangehas been given, and proceeds to step ST5. Meanwhile, if the driver'shead position does produce a predetermined motion, the displaycontroller 20 determines that an instruction to change the visible rangehas not been given, and proceeds to step ST6.

In step ST5, the display controller 20 conducts a process to change thevisible range. The display controller 20 moves the visible range that isvisible to the driver, according to the motion of the driver's headposition. The display controller 20 controls the backlight or brightnesslevel to move a darkly-displayed area according to the motion of thedriver's head position, for example, thereby changing thebrightly-displayed monitor image area. Also, the display controller 20additionally compresses the image in the non-monitor image area providedon the opposite side of the movement direction of the monitor imagearea, for example, thereby moving the visible range displayed by theimage in the monitor image area according to the motion of the driver'shead position. Additionally, since the range of the surrounding areadisplayed on the non-monitor image area provided in the movementdirection of the monitor image area becomes smaller in range, there isless compression of the non-monitor image area. The display controller20 conducts a process to change the visible range, and proceeds to stepST6.

In step ST6, the display controller 20 determines whether or not anoperation of checking the surrounding area has ended. The displaycontroller 20, on the basis of an image signal supplied from the driverimage capture unit 12, determines whether or not the driver's headorientation and line of sight direction are no longer in the directionof the mirror unit 55, for example. If the driver's head orientation andline of sight direction continue to be in the direction of the mirrorunit 55, the display controller 20 determines that an operation ofchecking the surrounding area has not ended, and returns to step ST4.Meanwhile, if the driver's head orientation and line of sight directionare no longer in the direction of the mirror unit 55, the displaycontroller 20 determines that an operation of checking the surroundingarea has ended, and proceeds to step ST7.

In step ST7, the display controller 20 ends the display. The displaycontroller 20 ends the image display of the surrounding area on thedisplay unit 50 to allow the driver to concentrate on driving, andreturns to step ST1.

FIG. 13 is a diagram for explaining a first display control operation ofa display controller. In the first display control operation, whiledisplaying a captured image of the surrounding area, the displaycontroller lowers the brightness of some or all of the image in theother range excluding the visible range, making the other range lessvisible to the driver. In so doing, by viewing an image via the mirrorunit 55, the driver becomes able to check the desired visible range ofthe surrounding area similarly to the case of using a rearview mirror.In addition, there are merits such as moderating unwanted brightness inthe non-monitor image area that produces glare, particularly at night.

FIG. 13(A) illustrates the display on the display unit 50, while FIG.13(B) illustrates the relationship between the display position and thebrightness of the display unit 50. The display controller 20 lowers thebrightness of the non-monitor image areas GRb1 and GRb2, which are theimage areas corresponding to the other range excluding the visiblerange, to lower than the brightness of the monitor image area GRmcorresponding to the visible range, making the images in the non-monitorimage areas GRb1 and GRb2 less visible. For example, if the display unit50 is configured using liquid crystal display elements, backlightemission control is conducted to darken the non-monitor image areas GRb1and GRb2. Meanwhile, if the display unit 50 is configured using organicEL display elements, signal level control of the luma signal isconducted to darken the non-monitor image areas GRb1 and GRb2. In thisway, in the display image of the display unit 50, since the non-monitorimage areas GRb1 and GRb2 are put into a dark state, the driver DRbecomes able to view only the image in the monitor image area GRm.Consequently, the driver DR is able to check the conditions of thevisible range ARm of the surrounding area by viewing the display imageof the display unit 50 via the mirror unit 55.

Also, when the display controller 20 determines that the head positionof the driver DR has moved towards the front of the vehicle, forexample, the display controller 20 moves the monitor image area GRmaccording to the movement of the head position, and moves the visiblerange of the surrounding area that is visible via the mirror unit 55outward. Specifically, the display controller 20 controls the brightnessof the image according to the movement of the head position towards thefront of the vehicle so that the area width of the non-monitor imagearea GRb1 becomes narrower, while the area width of the non-monitorimage area GRb2 becomes wider. By conducting such a display control, theimage in the monitor image area GRm becomes an image of a more outwardrange of the surrounding area compared to before the movement. When thedisplay controller 20 determines that the head position of the driver DRhas moved towards the rear of the vehicle, for example, although notillustrated, the display controller 20 controls the brightness of theimage according to the movement of the head position so that the areawidth of the non-monitor image area GRb1 becomes wider, while the areawidth of the non-monitor image area GRb2 becomes narrower. By conductingsuch a display control, the image in the monitor image area GRm becomesan image of a more inward range of the surrounding area compared tobefore the movement. As a result, by moving his or her head position,the driver is able to check the desired visible range of the surroundingarea similarly to the case of using a rearview mirror. In addition,since the display unit 50 and the mirror unit 55 are provided in thecabin of the vehicle, the surrounding area may be checked well, withoutbeing adversely affected by the side windows as in the case of using arearview mirror.

FIG. 14 is a diagram for explaining a second display control operationof a display controller. In the second display control operation, whiledisplaying a captured image of the surrounding area, instead of movingthe monitor image area GRm, the display controller compresses the imageof the other range excluding the visible range in the movement directionof the monitor image area GRm. By compressing the image in this way, thedisplay controller moves the visible range ARm corresponding to themonitor image area GRm, enabling the driver to check the desired visiblerange of the surrounding area.

FIG. 14(A) illustrates the display on the display unit 50, while FIG.14(B) illustrates the relationship between the display position and thecompression of the display unit 50, and FIG. 14(C) illustrates therelationship between the display position and the brightness of thedisplay unit 50. The display controller 20 conducts a display control sothat, for the non-monitor image areas GRb1 and GRb2 which are the imageareas corresponding to the other range excluding the visible range, theimage is reduced more towards the edges. By compressing the non-monitorimage areas GRb1 and GRb2 and adjusting the compression ratios of thenon-monitor image areas GRb1 and GRb2 in this way, the visible range ARmcorresponding to the monitor image area GRm becomes moveable.

When the display controller 20 determines from the motion of the headposition of the driver DR that an instruction to move the visible rangeof the surrounding area outward has been given, such as when the displaycontroller 20 determines that the driver has turned his or her neck inthe forward direction, for example, the display controller 20 moves thevisible range corresponding to the monitor image area GRm outward.Specifically, the display controller 20 controls the compression ratioof the image so that the range of the surrounding area for thenon-monitor image area GRb2 becomes wider. In addition, by reducing thedegree of compression and lowered brightness of the non-monitor imagearea GRb1 corresponding to the outward direction, the driver becomesable to check a more outward range than the visible range with the imagein the non-monitor image area GRb1. In other words, the visible rangemay be extended. Note that even if only the degree of lowered brightnessof the non-monitor image area GRb1 is reduced, the driver becomes ableto check a more outward range than the visible range, and the visiblerange may be extended.

FIG. 15 illustrates a relationship between the display and compressionratio of a display unit, and the surrounding area. Note that FIG. 15(A)illustrates the display on the display unit 50, while FIG. 15(B)illustrates the relationship between the display position and thecompression of the display unit 50, and FIG. 15(C) illustrates the imagecapture range of the surrounding area image capture unit 11. In themonitor image area GRm, a captured image of the visible range ARm isdisplayed. In the non-monitor image area GRb1, a captured image moreoutward than the visible range ARm is compressed and displayed, while inthe non-monitor image area GRb2, a captured image more inward than thevisible range ARm is compressed and displayed. Herein, when the displaycontroller 20 determines that an instruction to move the visible rangeoutward has been given, the display controller 20 controls thecompression ratio of the image so that the range of the surrounding areafor the non-monitor image area GRb2 becomes wider. Consequently, thevisible range ARm corresponding to the monitor image area GRm movesoutward in the surrounding area.

Meanwhile, when the display controller 20 determines that an instructionto move the visible range inward has been given, such as when thedisplay controller 20 determines that the driver has turned his or herneck in the rear direction, for example, the display controller 20controls the compression ratio of the image so that the range of thesurrounding area for the non-monitor image area GRb1 becomes wider.Consequently, the visible range ARm corresponding to the monitor imagearea GRm moves inward in the surrounding area.

Furthermore, as discussed earlier, before an instruction to change thevisible range is given, the display controller 20 lowers the brightnessof the non-monitor image areas GRb1 and GRb2 to lower than thebrightness of the monitor image area GRm corresponding to the visiblerange, making the images in the non-monitor image areas GRb1 and GRb2less visible. For example, if the display unit 50 is configured usingliquid crystal display elements, backlight emission control is conductedto darken the non-monitor image areas GRb1 and GRb2. Meanwhile, if thedisplay unit 50 is configured using organic EL display elements, signallevel control of the luma signal is conducted to darken the non-monitorimage areas GRb1 and GRb2. In this way, in the display image of thedisplay unit 50, since the non-monitor image areas GRb1 and GRb2 are putinto a dark state, the driver DR becomes able to view only the image inthe monitor image area GRm. Consequently, the driver DR is able to checkthe conditions of the visible range ARm of the surrounding area byviewing the display image of the display unit 50 via the mirror unit 55.After that, if an instruction to change the visible range is given, thedisplay controller 20 causes the brightness level of a non-monitor imagearea to approach the brightness level of the monitor image area GRm.

By conducting such display control, when the driver gives an instructionto change the visible range, the visible range ARm corresponding to themonitor image area GRm is moved according to the driver's instruction,thereby enabling the driver to easily check a desired range of thesurrounding area. Furthermore, since the brightness level of anon-monitor image area is made similar to the brightness level of themonitor image area GRm, the division from the monitor image area isremoved. Consequently, the visible range may be extended automatically.Furthermore, since the driver is able to change the visible range bysimply giving a change instruction, the driver becomes able to easilycheck a desired surrounding range, without moving his or her headposition in the forward direction or the like until the desiredsurrounding range is reflected in the rearview mirror, as in the case ofusing a rearview mirror of the past. In addition, since the display unit50 and the mirror unit 55 are provided in the cabin of the vehicle, thesurrounding area may be checked well, without being adversely affectedby the side windows as in the case of using a rearview mirror.

Note that when the driver gives an instruction to change the visiblerange, the brightness level of the non-monitor image area may also bekept in a lowered state. In this case, extension of the monitor imagearea GRm is not conducted, an image of the visible range that has beenmoved on the basis of the change instruction is displayed at apredetermined brightness. Also, the compression characteristics andbrightness characteristics of the non-monitor image areas are notlimited to the characteristics in FIG. 14. For example, the compressionratio may also be varied suddenly, whereas the brightness level may alsobe varied gradually. In addition, by making the compressioncharacteristics and the brightness characteristics selectable by thedriver, an image of the surrounding area may be displayed according tothe driver's preferences. Furthermore, properties such asenlargement/reduction of the visible range or the enlargement ratio ofthe displayed image may also be varied according to an instruction tochange the visible range. For example, an operation in which the driverDR moves or has moved closer to the mirror unit 55 may be treated as aninstruction to change the visible range, and an image with a widevisible range or an image with a low enlargement ratio (these images maybe designated a first image, for example) is displayed. Meanwhile, anoperation in which the driver DR moves or has moved away from the mirrorunit 55 may be treated as an instruction to change the visible range,and an image with a narrow visible range or an image with a highenlargement ratio compared to the first image is displayed.

3. Third Embodiment

The third embodiment describes a case in which the surrounding area thatthe driver wants to check changes depending on the driving conditions ofthe vehicle. As an example, a case will be described in which the cabinand a trailer part are separable, like with a trailer, an articulatedbus, or a camping car, for example, and in which the orientation of thecabin and the orientation of the trailer part change depending on thedriving conditions.

3-1. Configuration of Third Embodiment

FIG. 16 illustrates an example of a configuration of the thirdembodiment. A display device 10 is equipped with surrounding area imagecapture units 11 a and 11 b, a driver image capture unit 12, a drivingconditions detection sensor 13, a display controller 20, a display unit50, and a mirror unit 55. Note that the driver image capture unit 12,the driving conditions detection sensor 13, the display unit 50, themirror unit 55, and the driver DR exist in the positional relationshipillustrated in FIG. 9.

The surrounding area image capture units 11 a and 11 b capture an imageof the surrounding area around a vehicle, and output an image signal tothe display controller 20. The surrounding area image capture unit 11 ais provided on the cabin, while the surrounding area image capture unit11 b is provided on the trailer part. Herein, the area captured by thesurrounding area image capture unit 11 a will be designated the firstcaptured surrounding area, while the area captured by the surroundingarea image capture unit 11 b will be designated the second capturedsurrounding area.

The driver image capture unit 12 is provided in front of the driver DRor in the direction in which the mirror unit 55 is installed, forexample, to enable determination of properties such as the headposition, head orientation, and line of sight orientation of the driverDR. The driver image capture unit 12 captures an image of the driver DR,and outputs an image signal to the display controller 20.

The driving conditions detection sensor 13 acquires information relatedto the driving conditions, such as information indicating the steeringstate, information indicating the coupling state of the trailer part,information about the turning of the trailer part with respect to thecabin, gear position information and vehicle speed information,direction indicator configuration information, and current positioninformation (for example, a positioning signal of a satellitepositioning system), for example. The driving conditions detectionsensor 13 outputs the acquired driving information to the displaycontroller 20 as sensor information.

The display controller 20, on the basis of image signals generated bythe surrounding area image capture units 11 a and 11 b, displays acaptured image of the surrounding area on the display unit 50. Inaddition, the display controller 20 controls the display of a capturedimage of the surrounding area displayed on the display unit 50, on thebasis of properties such as the head position, head orientation, andline of sight orientation of the driver, movement of the position andorientation, and sensor information from the driving conditionsdetection sensor 13.

The display unit 50 is arranged so that the display face of the displayunit 50 is indirectly visible to the driver via the mirror unit 55.Also, the display face of the display unit 50 is configured to a sizelarger than the ray of light in which the mirror face of the mirror unit55 converges, as illustrated in FIG. 4, so that a display image of thedisplay unit 50 is visible via the mirror unit 55, even if the driver DRmoves his or her head position to check an area over a wide range with arearview mirror while driving, for example. Note that in a display imageof the display unit 50, the area that corresponds to the visible rangeof the image area that the driver checks via the mirror unit 55 will bereferred to as the monitor image area.

The mirror unit 55 is provided so that the display face of the displayunit 50 is indirectly visible to the driver DR. The mirror unit 55 isplaced inside the vehicle, at a position of a pillar, for example, sothat an image reflected in the mirror unit 55 is visible when the driverDR performs an operation of looking at a rearview mirror of the past,for example. In addition, the positional relationship of the mirror unit55 with the display unit 50 and the size of the mirror unit 55 areconfigured so that when the driver DR indirectly views a captured imageof the surrounding area via the mirror unit 55, the entire display areaof the display unit 50 is reflected in the mirror unit 55. Furthermore,in the display unit 50 and the mirror unit 55, the visible range of thesurrounding area that is visible to the driver from the image of thedisplay unit 50 reflected in the mirror unit 55 is changed by displaycontrol of the display controller 20, according to movement of theviewing position of the driver with respect to the mirror unit 55. Notethat the size of the mirror unit 55 is preferably configured as in thefirst embodiment and the second embodiment, so that operationaladvantages similar to a rearview mirror of the past are obtained.

For the display device 10, in order for a captured image of thesurrounding area to be indirectly visible to the driver DR via themirror unit 55, the distance from the driver DR to the display face ofthe display unit 50 is increased compared to the case of the driver DRdirectly viewing the display face on the display unit 50.

In addition, in the display device 10, since a captured image of thesurrounding area is indirectly visible to the driver DR via the mirrorunit 55, the display unit 50 is arranged so that the display face andilluminating light from the display face are not visible to the driverDR. In addition, a configuration provided with a shield that blocks viewof the display face of the display unit 50 and display light from thedriver DR may also be used.

FIG. 17 illustrates an example of the arrangement of surrounding areaimage capture units. Note that the drawing illustrates an example of thecase of arranging surrounding area image capture units on the left-sideface of the trailer, for example, while the surrounding area imagecapture units on the right-side face are omitted.

As discussed earlier, the surrounding area image capture unit 11 a isprovided on the cabin, while the surrounding area image capture unit 11b is provided on the trailer part. Herein, the area captured by thesurrounding area image capture unit 11 a will be designated the firstcaptured surrounding area, while the area captured by the surroundingarea image capture unit 11 b will be designated the second capturedsurrounding area.

FIG. 17(A) illustrates a case in which the cabin and the trailer partare positioned in a straight line. FIG. 17(B) illustrates a state inwhich the trailer part is veering to the side where the surrounding areaimage capture unit 11 a is installed with respect to the cabin, and FIG.17(C) illustrates a state in which the trailer part is veering in theopposite direction from the side where the surrounding area imagecapture unit 11 a is installed with respect to the cabin. If the trailerpart goes from the state illustrated in FIG. 17(A) to the stateillustrated in FIG. 17(B), an image portion depicting the trailer partis added to the captured image acquired by the surrounding area imagecapture unit 11 a, and the driver may no longer be able to check thesurrounding area. Also, if the trailer part goes from the stateillustrated in FIG. 17(A) to the state illustrated in FIG. 17(C), thetrailer part departs from the image capture range of the surroundingarea image capture unit 11 a, and the vicinity of the trailer partbecomes a blind spot that the driver is unable to check. On the otherhand, since the surrounding area image capture unit 11 b is provided onthe trailer part, the captured image acquired by the surrounding areaimage capture unit 11 b becomes an image illustrating the surroundingarea irrespectively of the positional relationship between the cabin andthe trailer part.

Accordingly, the display controller 20 uses the captured image acquiredby the surrounding area image capture unit 11 a and the captured imageacquired by the surrounding area image capture unit 11 b depending onthe driving conditions and the driver's intentions, and thereby controlsthe display so that the surrounding area may be checked via the mirrorunit 55.

FIG. 18 is a diagram illustrating a configuration of a displaycontroller. The display controller 20 is equipped with a driver movementdetermination unit 21, a driving conditions determination unit 22, acontrol processor 35, a display adjuster 41, and a brightness adjuster42.

The driver movement determination unit 21, on the basis of an imagesignal supplied from the driver image capture unit 12, detects the headposition of the driver, and determines the movement direction andmovement magnitude of the head position of the driver. For example, thedriver movement determination unit 21 conducts facial recognition of thedriver on the basis of an image signal supplied from the driver imagecapture unit 12, for example, and determines the position of therecognized face, the orientation of the face or the orientation of thehead. Furthermore, the driver movement determination unit 21 tracks therecognized face, and determines the movement direction and movementmagnitude of the head position. The driver movement determination unit21 outputs a determination result to the control processor 35.

The driving conditions determination unit 22 determines the drivingconditions on the basis of the sensor information supplied from thedriving conditions detection sensor 13. The driving conditionsdetermination unit 22 determines whether the vehicle is moving forwardor backward on the basis of the gear position and the like, and whetherthe vehicle is moving straight, turning right, or turning left, on thebasis of the vehicle speed, direction indicator configurationinformation, the steering state, and the like, for example. In addition,the driving conditions determination unit 22 determines factors such aswhether there is a gentle left/right turn or a sharp left/right turn,and whether or not the travel position is at the position of aroundabout, on the basis of information about the coupling state of thetrailer part, the turning of the trailer part, and the like, forexample. The driving conditions determination unit 22 outputs adetermination result to the control processor 35.

The control processor 35, on the basis of a determination result fromthe driver movement determination unit 21 and the driving conditionsdetermination unit 22, generates a control signal to control the displaydifferently for the monitor image area and the non-monitor image area ofthe display unit 50, and outputs to the display adjuster 41 and thebrightness adjuster 42.

The display adjuster 41, on the basis of a control signal from thecontrol processor 35, conducts a scale adjustment of the captured imageof the surrounding area on an image signal supplied from the surroundingarea image capture unit 11, such as by applying image compression to thenon-monitor image area, for example. In addition, the display adjuster41, on the basis of a control signal from the control processor 35, mayalso switch or combine images of the surrounding area acquired by aplurality of surrounding area image capture units, adjust the displaywidth of the surrounding area to display, and the like.

The brightness adjuster 42, on the basis of a control signal from thecontrol processor 35, lowers the brightness of the non-monitor imagearea on the display unit 50 to less than the monitor image area. If thedisplay unit 50 is configured using display elements that may requireillumination, such as liquid crystal elements, for example, thebrightness adjuster 42 controls the illumination, such as the backlight,for example, to lower the brightness of the non-monitor image area toless than the monitor image area. Meanwhile, if the display unit 50 isconfigured using display elements that do not require illumination orusing light-emitting elements, such as organic EL display elements, forexample, a process may be conducted to lower the signal level of a lumasignal corresponding to the non-monitor image area.

3-2. Operation of Third Embodiment

FIG. 19 is a flowchart illustrating operation of a display controller inthe third embodiment. In step ST11, the display controller 20 determineswhether or not an operation of checking the surrounding area has beenperformed. The display controller 20, on the basis of an image signalsupplied from the driver image capture unit 12, determines whether ornot the driver's head orientation and line of sight direction are in thedirection of the mirror unit 55, for example. If the driver's headorientation and line of sight direction are in the direction of themirror unit 55, such as when the driver glances in the direction of themirror unit 55, for example, the display controller 20 determines thatan operation of checking the surrounding area has been performed, andproceeds to step ST12. Meanwhile, if the driver's head orientation andline of sight direction are not in the direction of the mirror unit 55,the display controller 20 determines that an operation of checking thesurrounding area has not been performed, and returns to step ST11.

In step ST12, the display controller 20 determines the drivingconditions. The display controller 20 determines the driving conditionson the basis of the sensor information supplied from the drivingconditions detection sensor 13. The display controller 20 determinesfactors such as whether the vehicle is moving forward or backward,whether the vehicle is moving straight, turning right, or turning left,whether or not there is a gentle left/right turn or a sharp left/rightturn, and whether the travel position is at the position of aroundabout, for example, and proceeds to step ST13.

In step ST13, the display controller 20 determines the visible range.The display controller 20 determines the visible range that is visiblevia the mirror unit 55 on the basis of the driving conditionsdetermination result, and proceeds to step ST14. Note that in thedetermination of the visible range, the visible range may be determinedby additionally accounting for the head position of the driver.

FIG. 20 illustrates an example of visible range modes. Mode 1 is amirror visibility base range during normal travel, for example. Mode 2is a range enabling wider visibility both inward and outward from themirror visibility base range during normal travel, for example. Mode 3is configured as a range enabling the driver to view the area outwardfrom the vehicle with no blind spots when backing up or the like, forexample. Herein, suppose that the image capture range of the surroundingarea image capture unit 11 a is the mirror visibility base range duringnormal travel, while the image capture range of the surrounding areaimage capture unit 11 b is a wider range than the image capture range ofthe surrounding area image capture unit 11 a, for example. In addition,as illustrated in FIG. 20(A), suppose that Mode 1 treats the imagecapture range of the surrounding area image capture unit 11 a as thevisible range, while Mode 2 treats the image capture range of thesurrounding area image capture unit 11 b as the visible range.Furthermore, as illustrated in FIG. 20(B), suppose that Mode 3 treatsthe range included in the image capture ranges of the surrounding areaimage capture unit 11 a and the surrounding area image capture unit 11 bas the visible range. Note that the image capture range of thesurrounding area image capture unit 11 a may also be switchable, and theimage capture range of the surrounding area image capture unit 11 b maybe treated as the visible range depending on the driving conditions or adriver instruction. For example, as illustrated in FIG. 20(C), the imagerange of the surrounding area image capture unit 11 a may also beswitchable to the ranges of Mode 1 to Mode 3.

FIG. 21 illustrates an example of a relationship between drivingconditions determination results and visible range modes. When thedisplay controller 20 determines on the basis of detected informationthat the vehicle is being driven straight ahead, for example, the rangeof Mode 1 is treated as the visible range, as indicated by the circlesymbol. When the display controller 20 determines on the basis ofdetected information that the vehicle is at the location of aroundabout, for example, the range of Mode 1 is treated as the visiblerange, as indicated by the circle symbol. When the display controller 20determines on the basis of detected information that the vehicle ismaking a gentle right turn or left turn, for example, the range of Mode1 is treated as the visible range, as indicated by the circle symbol.When the display controller 20 determines on the basis of detectedinformation that the vehicle is making a sharp right turn or left turn,for example, the range of Mode 2 is treated as the visible range and thecheckable range is widened outward, as indicated by the circle symbol.When the display controller 20 determines on the basis of detectedinformation that the vehicle is backing up straight, for example, therange of Mode 1 is treated as the visible range, as indicated by thecircle symbol. When the display controller 20 determines on the basis ofdetected information that the vehicle is backing up at a sharp angle,for example, the range of Mode 2 indicated by the circle symbol, or inother words, the image capture range of the surrounding area imagecapture unit 11 b, is treated as the visible range so that the blindspot illustrated in FIG. 17(C) does not occur.

In step ST14, the display controller 20 conducts an image displaycontrol process. The display controller 20 configures the area width ofthe monitor image area according to the visible range mode, and displaysan image of the visible range on the display unit 50 as an image in themonitor image area. In addition, the display controller 20 controls thebacklight or the brightness level, for example, so that an image in thenon-monitor image area is not visible. In addition, when treating apredetermined area of the display unit 50 as a monitor image area, thedisplay controller 20 conducts a process such as compressing the imageof the surrounding area corresponding to the non-monitor image area, sothat an image of the visible range may be displayed in the monitor imagearea, and also so that the visible range corresponding to the monitorimage area may be moved. The display controller 20 conducts an imagedisplay control process, and proceeds to step ST15.

In step ST15, the display controller 20 determines whether or not aninstruction to change the visible range has been given. The displaycontroller 20, on the basis of an image signal supplied from the driverimage capture unit 12, determines the driver's head position, forexample, and determines whether or not an instruction operation has beenperformed to change the visible range of the surrounding area. If thedriver's head position produced a motion with movement of the visiblerange, the display controller 20 determines that an instruction tochange the visible range has been given, and proceeds to step ST16.Meanwhile, if the driver's head position does produce a motion withmovement of the visible range, the display controller 20 determines thatan instruction to change the visible range has not been given, andproceeds to step ST17.

In step ST16, the display controller 20 conducts a mode change process.The display controller 20 switches to a mode with a wide visible rangewhen the instruction to change the visible range is an instruction towiden the visible range. For example, when set to a mode indicated by acircle symbol in FIG. 21, the display controller 20 switches to a modeindicated by a square symbol, and when set to a mode indicated by asquare symbol, the display controller 20 switches to a mode indicated bya star symbol. Also, the display controller 20 switches to a mode with anarrow visible range when the instruction to change the visible range isan instruction to narrow the visible range. For example, when set to amode indicated by a square symbol in FIG. 21, the display controller 20switches to a mode indicated by a cycle symbol, and when set to a modeindicated by a star symbol, the display controller 20 switches to a modeindicated by a square symbol. The display controller 20 switches themode on the basis of the instruction to change the visible range, andproceeds to step ST17.

In step ST17, the display controller 20 determines whether or not anoperation of checking the surrounding area has ended. The displaycontroller 20, on the basis of an image signal supplied from the driverimage capture unit 12, determines whether or not the driver's headorientation and line of sight direction are no longer in the directionof the mirror unit 55, for example. If the driver's head orientation andline of sight direction continue to be in the direction of the mirrorunit 55, the display controller 20 determines that an operation ofchecking the surrounding area has not ended, and returns to step ST15.Meanwhile, if the driver's head orientation and line of sight directionare no longer in the direction of the mirror unit 55, the displaycontroller 20 determines that an operation of checking the surroundingarea has ended, and proceeds to step ST18.

In step ST18, the display controller 20 ends the display. The displaycontroller 20 ends the image display of the surrounding area on thedisplay unit 50 to allow the driver to concentrate on driving, andreturns to step ST11.

By conducting such display control, the configuration and switching ofthe visible range may be conducted automatically according to thedriving conditions, enabling the driver to check a desired visible rangeof the surrounding area more easily than in the case of using a rearviewmirror of the past. For example, when the travel direction of thevehicle is straight ahead on the basis of sensor information suppliedfrom the driving conditions detection sensor 13, the mode is switchedfrom Mode 1 to Mode 2, and the visible range is extended in the outwarddirection of the vehicle. Also, in the case of a sharp left/right turnor backing up at a sharp angle in which the orientation of the cabin andthe orientation of the trailer part are different on the basis of sensorinformation supplied from the driving conditions detection sensor 13,the mode is switched from Mode 2 to Mode 3, and the visible range isextended so that the outward side of the trailer part is included in thevisible range. Consequently, the desired visible range may be checked.In addition, since the display unit 50 and the mirror unit 55 areprovided in the cabin of the vehicle, the surrounding area may bechecked well, without being adversely affected by the side windows as inthe case of using a rearview mirror.

FIG. 22 illustrates an example of a display image displayed on thedisplay unit 50 using surrounding images acquired by the surroundingarea image capture unit 11 a and the surrounding area image capture unit11 b. FIG. 22(A) illustrates an example of a case in which a car ispositioned in the blind spot of the surrounding area image capture unit11 a. FIG. 22(B) illustrates an example of a case in which a surroundingimage MGa acquired by the surrounding area image capture unit 11 a isdisplayed, in which the entire vehicle behind may not be checked withonly the display image of an image captured by the surrounding areaimage capture unit 11 a. For this reason, as illustrated in FIG. 22(C),the display controller 20 displays not only the surrounding image MGaacquired by the surrounding area image capture unit 11 a, but alsodisplays a surrounding image MGb acquired by the surrounding area imagecapture unit 11 b, thereby enabling the driver to check the entirevehicle behind. Note that if the display controller 20 is configured toswitch the display illustrated in FIGS. 22(B) and 22(C) or vary the areawidth (or insertion width) of the surrounding image MGa and thesurrounding image MGb as indicated by the arrows in FIG. 22(C) accordingto the driving conditions or the driver's intentions, the surroundingarea may be displayed in the most easily visible state.

FIG. 23 illustrates another example of a display image displayed on thedisplay unit 50 using surrounding images acquired by the surroundingarea image capture unit 11 a and the surrounding area image capture unit11 b. FIG. 23(A) illustrates an example of a case in which a surroundingimage MGa acquired by the surrounding area image capture unit 11 a isdisplayed. FIG. 23(B) illustrates an example of a case in which asurrounding image MGa acquired by the surrounding area image captureunit 11 a and a surrounding image MGb acquired by the surrounding areaimage capture unit 11 b are displayed by being arranged vertically. Thedisplay controller 20 switches between the display illustrated in FIG.23(A) and the display illustrated in FIG. 23(B) according to the drivingconditions and the driver's intentions. In addition, a configurationable to vary the area width (or insertion width) between the surroundingimage MGa and the surrounding image MGb as indicated by the arrows inFIG. 23(B) may also be used. At this point, when presenting a display onthe display unit with adjacent image displays captured by plural imagecapture, captured images arranged farther to the front with respect tothe travel direction of the vehicle body may be arranged higher on thedisplay screen, whereas captured images installed farther to the rearwith respect to the travel direction may be arranged lower on thedisplay screen. Such an arrangement has the merit of enabling the driverto intuitively and instantaneously determine the front/rear relationshipof the vehicle body on the visible screen. In other words, when at leasttwo or more image capture devices are attached on the vehicle exteriorand facing to the rear with respect to the vehicle travel direction, anda display unit provided with at least two or more adjacently arrangedscreen display areas is included in the cabin of the vehicle, such as inthe dashboard center, for example, in the displayed content of thecaptured images to display in the screen display areas, the image from afirst image capture device (the surrounding area image capture unit 11a, for example) installed on the front side in the vehicle traveldirection is arranged above the image from an image capture device (thesurrounding area image capture unit 11 b, for example) installed fartherto the rear in the vehicle travel direction than the first image capturedevice (for example, FIG. 23(B)). Such a configuration has a merit ofenabling the driver to intuitively and instantaneously determine thefront/rear relationship of the vehicle body on the visible screen.

In addition, even with a layout in which the screens are large and avertical arrangement is unavailable, the rear vanishing point of thescreen of the image capture unit installed to the front in the traveldirection is arranged on top. In other words, when at least two or moreimage capture devices are attached on the vehicle exterior and facing tothe rear with respect to the vehicle travel direction, and a displayunit provided with at least two or more adjacently arranged screendisplay areas is included in the cabin of the vehicle, in the displayedcontent of the captured images to display in the screen display areas,the vanishing point at infinity included in the image from the firstimage capture device installed on the front side in the vehicle traveldirection is arranged above the vanishing point at infinity included inthe image from the image capture device installed farther to the rear inthe vehicle travel direction than the first image capture device.According to such a configuration, similar advantageous effects areobtained. In this case, a shift in the vertical direction of at least20% or more of the display screens is desirable to more rapidlydistinguish the difference.

FIG. 24 illustrates another example of a display image displayed on thedisplay unit 50 using surrounding images acquired by the surroundingarea image capture unit 11 a and the surrounding area image capture unit11 b. FIG. 24(A) illustrates an example of a case in which a surroundingimage MGa acquired by the surrounding area image capture unit 11 a isdisplayed. FIG. 24(B) illustrates an example of a case in which asurrounding image MGa acquired by the surrounding area image captureunit 11 a and a surrounding image MGb acquired by the surrounding areaimage capture unit 11 b are displayed by being arranged vertically, andin which is provided a display that enables the driver to identify whichsurrounding area image capture unit acquired each surrounding image,such as an icon display HT that diagrammatically illustrates the cabinand the trailer part, for example. Furthermore, on the icon display HT,a camera-shaped mark HC indicating the surrounding area image captureunit that acquired the surrounding image displayed adjacently isprovided, for example. In this way, by arranging a display imagecaptured by the surrounding area image capture unit 11 a disposed at thefront of the vehicle on top, and arranging a display image captured bythe surrounding area image capture unit 11 b disposed at the rear of thevehicle on the bottom, the driver becomes able to easily grasp thecorrespondence relationship between the displayed surrounding images andthe surrounding area image capture units, and becomes able to referencethe display images and perform appropriate driving operations. Thedisplay controller 20 switches between the display illustrated in FIG.24(A) and the display illustrated in FIG. 24(B) according to the drivingconditions and the driver's intentions. In addition, a configurationable to vary the area width (or insertion width) between the surroundingimage MGa and the surrounding image MGb as indicated by the arrows inFIG. 24(B) may also be used.

Also, in cases such as normal travel, only the surrounding imageacquired by the surrounding area image capture unit 11 a is displayed,for example, and thus the obstruction of the visual field of the frontalperipheral area by displaying an arrangement of a plurality ofsurrounding images may be avoided.

In addition, the switching of the surrounding area image capture unitand the control of the surrounding image to display varies the displaycontent according to the driving state of the vehicle, while alsoenabling appropriate switching based on the driver's intentions.Consequently, the driver is able to intuitively and naturally grasp thevisible area in each area without carefully observing the screen. In thecase of a tow vehicle such as a tractor-trailer, a display that enablesinstantaneous per-section screen recognition with boundaries, ratherthan a combined screen display on a single screen, is desirable from anergonomic perspective, and thus the display controller 20 presents anergonomically suitable display.

In addition, a condition in which the blind spot becomes important in atow vehicle such as a tractor-trailer is during small-curvature travel,such as when changing lanes to the outer lane in a roundabout with twoor more lanes, or when checking for rear obstacles while backing up, forexample. Accordingly, by conducting a screen transition display thatreflects the intended display of the driver's screen operations inconjunction with the driving state of the vehicle, the grasp ofconditions in the blind spot may be improved. In such conditions, ratherthan changing the visual field area of a single camera, an operationthat switches or varies the display area after switching to the cameraof the surrounding area image capture unit 11 b on the trailer part iseffective.

In addition, for display extension, a display area extension functionthat is not limited to methods via the mirror unit 55 is effective.

Note that although the third embodiment describes the case of switchingmodes, the third embodiment may also be combined with the processingoperations of the second embodiment.

In addition, although the foregoing embodiments described the case ofconducting head orientation and point of view recognition, touch buttoncontrol, voice command recognition, non-head-based gesture recognition,and the like may also be conducted as an auxiliary system configuration.Particularly, when restoring a normal state, correcting misdetections ofinstruction content recognition during learning as discussed later, andalso when the driver performs a complex change of posture during alow-speed parking operation, it is effective to conduct control incombination with an interface that gives direct instructions, such as bybutton operations.

4. Fourth Embodiment

When viewing a surrounding area using a display device, a fixed screenscale is ensured during normal travel, in accordance with a minimumdisplay scale determined by law. In other words, there is a displayscreen for a steady-state head orientation while driving, and when thehead orientation changes from the steady-state position to a differentposition, a display change instruction is received via a human-machineinterface for a declaration of intent that the driver treats asnon-steady-state, and a screen display change is conducted. For example,a screen display change is conducted after receiving an instruction suchas an instruction to change the display content to a lower scale, or aninstruction to change to a special field of view for non-steady-statetravel, such as when parking. Accordingly, the fourth embodimentdescribes the mechanism of the human-machine interface related tochanging the display content.

FIG. 25 is a flowchart illustrating an example of operation in a case inwhich the driver grasps the conditions of the surrounding area. In stepST21, the driver starts a glance operation, and proceeds to step ST22.In step ST22, the driver captures the mirror unit with his or her eyes,and proceeds to step ST23. In step ST23, the driver temporarily stops ornearly stops his or her head, and proceeds to step ST24. Note that“nearly stops” refers to a state of little head motion that may beinterpreted as a stopped state.

In step ST24, the driver focuses on an image via the mirror unit andproceeds to step ST25. In step ST25, the driver grasps the conditions ofthe surrounding area by checking the image of the visible range, andproceeds to step ST26.

In step ST26, the driver judges whether or not to change the visiblerange. If the driver has grasped the conditions of the desired range ofthe surrounding area by viewing the image in the monitor image area instep ST25, the driver decides to not change the visible range, andproceeds to step ST27. On the other hand, if the driver was unable tograsp the conditions of the desired range of the surrounding area, thedriver decides to change the visible range, and proceeds to step ST28.

In step ST27, the driver returns to the forward viewing state. Since thedriver was able to grasp the conditions of the desired surrounding area,the driver ends the glance and points his or her face forward, endingthe process in a state enabling the driver to look ahead.

If the process proceeds from step ST26 to step ST28, the driver gives aninstruction to change the visible range. The driver performs apredefined motion, such as a motion of repeatedly moving his or hertorso, for example, and proceeds to step ST29. Meanwhile, on the displaydevice, the driver's instruction to change the visible range isdetected, and a process of changing the visible range displayed by animage in the monitor image area is conducted. In the change of thevisible range, the range of the surrounding area visible to the driveris moved or extended.

In step ST29, the driver grasps the conditions of the surrounding areaby checking the image with the changed visible range, and proceeds tostep ST30.

In step ST30, the driver decides whether or not it is unnecessary torevert to the state before the change instruction. If the driver wantsto check the visible range from before the change instruction, thedriver decides to revert to the state before the change instruction, andproceeds to step ST31. On the other hand, if the driver decides to notrevert to the state before the change instruction, the driver proceedsto step ST32.

In step ST31, the driver gives an instruction to restore the visiblerange. The driver performs a predefined motion, such as a motion ofmoving his or her head back, for example, and proceeds to step ST32.Meanwhile, on the display device, the driver's instruction to restorethe visible range is detected, and a process of reverting the visiblerange displayed by an image in the monitor image area to the rangebefore the change is conducted.

In step ST32, the driver does not gaze at the mirror unit for a fixedperiod, and proceeds to step ST33.

In step ST33, the driver returns to the forward viewing state. Thedriver ends the grasping of the conditions of the surrounding area. Inother words, the driver ends the glance and points his or her faceforward, resulting in a state enabling the driver to look ahead.

The display device uses such a human-machine interface responsive todriver operations to change the visible range of the surrounding area onthe basis of a display change instruction from the driver. Note that adisplay device that uses such a human-machine interface is not limitedto a display device configured according to the first to thirdembodiments, and may also be applied to cases such as when the driverdirectly views the display on the display unit 50 to grasp theconditions of the surrounding area.

For the sake of simplicity, the flowchart illustrated in FIG. 25describes a series of operations being performed until returning tolooking ahead, in which the process simply returns directly. In actualpractice, more complex operations may be performed before returning tolooking ahead, and the phenomenon of the repeatedly checking by directviewing and viewing of the display unit before returning to lookingahead may occur many times. However, since the goal is not to describeevery possible phenomenon, description of examples other than the abovewill be omitted.

Next, configuration and operation according to the fourth embodimentwill be described for the case of the driver viewing an image on adisplay unit indirectly via a mirror unit, similarly to the first to thethird embodiments.

4-1. Configuration of Fourth Embodiment

FIG. 26 is a diagram illustrating an example of a configurationaccording to the fourth embodiment. A display device 10 is equipped witha surrounding area image capture unit 11, a driver image capture unit12, a driver identification information acquisition unit 15, a displaycontroller 20, a display unit 50, and a mirror unit 55. In addition, thedriver, the display unit, the mirror unit, and the driver image captureunit are provided as illustrated in FIG. 9 discussed previously.

The surrounding area image capture unit 11 captures an image of thesurrounding area around a vehicle, and output an image signal to thedisplay controller 20. Herein, the area captured by the surrounding areaimage capture unit 11 will be designated the captured surrounding area.

The driver image capture unit 12 is provided in front of the driver DRor in the direction in which the mirror unit 55 is installed, forexample, to enable determination of properties such as the headposition, head orientation, and line of sight orientation of the driverDR. The driver image capture unit 12 captures an image of the driver DR,and outputs an image signal to the display controller 20.

The driver identification information acquisition unit 15 acquiresdriver identification information, which is identification informationunique to the driver, and outputs to the display controller 20. Thedriver identification information acquisition unit 15 may use a driverface recognition obtained by the driver image capture unit 12, useidentification information assigned to a vehicle ignition key possessedby the driver, or implement various methods such as having the driverinput an instruction directly via button operations or the like.

The display controller 20 causes the display unit 50 to display acaptured image of the surrounding area that has been captured by thesurrounding area image capture unit 11. In addition, the displaycontroller 20 determines a glance in the direction of the mirror unit55, an operation of moving the area to check, and various instructionoperations, on the basis of properties such as the head position, headorientation, and line of sight orientation of the driver, movement ofthe position and orientation, and information supplied from the driveridentification information acquisition unit 15. Furthermore, the displaycontroller 20, on the basis of a determination result, controls thedisplay of a captured image of the surrounding area to be displayed onthe display unit 50. The display controller 20 displays a captured imageof the surrounding area on the display unit 50 when the driver glancesin the direction of the mirror unit, for example. In addition, thedisplay controller 20 extends the area of the visible range in the caseof determining that the driver has performed a predefined motion after aglance is detected, for example.

The display unit 50 is arranged so that the display face of the displayunit 50 is indirectly visible to the driver via the mirror unit 55.Also, the display face of the display unit 50 is configured to a sizelarger than the mirror face of the mirror unit 55, so that a displayimage of the display unit 50 is visible via the mirror unit 55, even ifthe driver DR moves his or her head position to check an area over awide range with a rearview mirror while driving. Note that in a displayimage of the display unit 50, the area that corresponds to the visiblerange of the image area that the driver checks via the mirror unit 55will be referred to as the monitor image area.

The mirror unit 55 is provided so that the display face of the displayunit 50 is indirectly visible to the driver DR. The mirror unit 55 isplaced inside the vehicle, so that an image reflected in the mirror unit55 is visible when the driver DR performs an operation of looking at arearview mirror of the past, for example. In addition, the positionalrelationship of the mirror unit 55 with the display unit 50 and the sizeof the mirror unit 55 are configured so that when the driver DRindirectly views a captured image of the surrounding area via the mirrorunit 55, the entire display area of the display unit 50 is reflected inthe mirror unit 55. Furthermore, in the display unit 50 and the mirrorunit 55, the visible range of the surrounding area that is visible tothe driver from the image of the display unit 50 reflected in the mirrorunit 55 is changed by display control of the display controller 20,according to movement of the viewing position of the driver with respectto the mirror unit 55. Note that the size of the mirror unit 55 ispreferably configured to make the size of the area depicting the visiblerange similar to the size of a rearview mirror of the past, so thatoperational advantages similar to a rearview mirror of the past areobtained.

For the display device 10, in order for a captured image of thesurrounding area to be indirectly visible to the driver DR via themirror unit 55, the distance from the driver DR to the display face ofthe display unit 50 is increased compared to the case of the driver DRdirectly viewing the display face on the display unit 50.

In addition, in the display device 10, since a captured image of thesurrounding area is indirectly visible to the driver DR via the mirrorunit 55, the display unit 50 is arranged so that the display face andilluminating light from the display face are not visible to the driverDR. In addition, a configuration provided with a shield that blocks viewof the display face of the display unit 50 and illumination light fromthe driver DR may also be used.

FIG. 27 is a diagram illustrating a configuration of a displaycontroller. The display controller 20 is equipped with a recognitionunit 23, a glance determination unit 24, an instruction operationdetermination unit 25, a driver authentication unit 26, a controlprocessor 35, a display adjuster 41, and a brightness adjuster 42.

The recognition unit 23 conducts face recognition on the basis of animage signal supplied from the driver image capture unit 12. Inaddition, the recognition unit 23 recognizes the orientation of therecognized head or the line of sight direction in the recognized face,and outputs a recognition result to the glance determination unit 24.

The glance determination unit 24 includes a glance determinationprocessor 241, a glance determination learning unit 242, and adetermination reference value storage unit 243.

The glance determination processor 241 compares a recognition resultfrom the recognition unit 23 to a glance determination reference valuesupplied from the glance determination learning unit 242, determineswhether or not the driver is facing in the direction of the mirror unit55, and outputs a determination result to the instruction operationdetermination unit 25 and the control processor 35.

The glance determination learning unit 242 reads out a glancedetermination reference value from the determination reference valuestorage unit 243, and outputs to the glance determination processor 241.In addition, the glance determination learning unit 242 reads out fromthe determination reference value storage unit 243 a glancedetermination reference value corresponding to factors such as driverinformation from the control processor 35, determination parametersettings for driving optimized for high-speed forward travel, anddetermination parameter settings for driving suited to low speed,backing up, or parking, and outputs to the glance determinationprocessor 241. Furthermore, the glance determination learning unit 242updates the glance determination reference value on the basis of therecognition result of the head orientation and the line of sightdirection, thereby enabling the glance determination processor 241 tomake a precise glance determination for each driver. In addition, sincean intended operation may differ even for similar head operationsdepending on the speed range of the vehicle, the glance determinationprocessor 241 is configured to make a determination dependent on thespeed range for each driver. For example, for high-speed travel orhigh-speed merging, the determination threshold time until confirming aline-of-sight glance may be set to less than one second to enable thedriver to pay primary attention ahead, and extend the visible rangewhile momentarily looking at the monitor with a fast operation.Meanwhile, since the respective actions for a large-sized trailer or thelike during cases such as parking are conducted by extension or otheroperations after first focusing one's line of sight on each state andgrasping the rear situation over a time that allows a full grasp, thedetermination threshold time is set to at least 0.5 seconds or more forspeeds of less than 15 km/h which are determined to correspond toparking. The glance determination learning unit 242 outputs an updatedglance determination reference value to the determination referencevalue storage unit 243, and updates the glance determination referencevalue being stored in the determination reference value storage unit243. Note that if a glance determination reference value correspondingto the reported driver is not stored in the determination referencevalue storage unit 243, the glance determination learning unit 242outputs a preconfigured glance determination reference value to theglance determination processor 241. In addition, the glancedetermination learning unit 242 causes the subsequently updated glancedetermination reference value to be stored in the determinationreference value storage unit 243 in association with the driverinformation. At this point, the determination trend of the headorientation recognition may be provided to the driver DR as visual oraudible feedback via a notification unit, such as an overlay display onthe display of the display unit 50, an LED display, or a speaker in thevehicle, for example. In so doing, it is possible to realizeconfirmation of an instruction operation more accurately and with aminimum level of operation. It is desirable for the feedback to thedriver DR to be not a digital 0/1, true or false determination, butrather analog feedback by the notification unit of a status indicatingthe degree of determination of the driver's motion, for example.

The determination reference value storage unit 243 stores a glancedetermination reference value used for glance determination. Inaddition, when driver authentication is conducted, the determinationreference value storage unit 243 stores a glance determination referencevalue for each driver. The glance determination reference value storedin the determination reference value storage unit 243 is updatedaccording to a learning result from the glance determination learningunit 242.

The instruction operation determination unit 25 includes an instructionoperation determination processor 251, an instruction operationdetermination learning unit 252, and a determination reference valuestorage unit 253.

The instruction operation determination processor 251 uses a recognitionresult from the recognition unit 23, a glance determination result fromthe glance determination unit 24, and a determination reference valuesupplied from the instruction operation determination learning unit 252to determine whether or not the driver performed a predeterminedoperation, and outputs to the control processor 35. The instructionoperation determination processor 251 determines an instruction by thedriver from the driver's motion, on the basis of a detection result of acombination of two or more acceleration or deceleration operations ofthe driver's head, for example.

The instruction operation determination learning unit 252 reads out aninstruction operation determination reference value from thedetermination reference value storage unit 253, and outputs to theinstruction operation determination processor 251. In addition, when adriver is reported from the control processor 35, the instructionoperation determination learning unit 252 reads out from thedetermination reference value storage unit 253 an instruction operationdetermination reference value corresponding to the reported driver, andoutputs to the instruction operation determination processor 251.Furthermore, the instruction operation determination learning unit 252updates the instruction operation determination reference value on thebasis of the recognition result of the head orientation and the line ofsight direction, thereby enabling the instruction operationdetermination processor 251 to make a precise instruction operationdetermination. The instruction operation determination learning unit 252outputs an updated instruction operation determination reference valueto the determination reference value storage unit 253, and updates theinstruction operation determination reference value being stored in thedetermination reference value storage unit 253. Note that if aninstruction operation determination reference value corresponding to thereported driver is not stored in the determination reference valuestorage unit 253, the instruction operation determination learning unit252 outputs a preconfigured instruction operation determinationreference value to the instruction operation determination processor251. In addition, the instruction operation determination learning unit252 causes the subsequently updated instruction operation determinationreference value to be stored in the determination reference valuestorage unit 253 in association with the driver information.

The determination reference value storage unit 253 stores an instructionoperation determination reference value used for instruction operationdetermination. In addition, when driver authentication is conducted, thedetermination reference value storage unit 253 stores an instructionoperation determination reference value for each driver. The instructionoperation determination reference value stored in the determinationreference value storage unit 253 is updated according to a learningresult from the instruction operation determination learning unit 252.

At this point, a function may be provided so that if an operationperformed with a gesture produces an unintended determination result inthe instruction operation determination learning unit 252 and leads toincorrect operation, a specification of an excluded category or the likemay be conducted in a feedback format other than gesture detection, suchas by a button operation or a speech operation, for example.

The driver authentication unit 26 determines the current driver of thevehicle on the basis of identification information unique to the driverthat is acquired by the driver identification information acquisitionunit 15, and outputs a determination result to the control processor 35.

The control processor 35 determines the driver of the vehicle on thebasis of a determination result from the driver authentication unit 26,and reports the determined driver to the glance determination learningunit 242 and the instruction operation determination learning unit 252.In addition, the control processor 35 generates a control signal thatcontrols the display differently for the monitor image area and thenon-monitor image area of the display unit 50, on the basis ofdetermination results from the glance determination unit 24 and theinstruction operation determination unit 25. The control processor 35outputs the generated control signal to the display adjuster 41 and thebrightness adjuster 42.

The display adjuster 41, on the basis of a control signal from thecontrol processor 35, conducts a scale adjustment of the captured imageof the surrounding area, changes or combines images of the surroundingarea, and the like with respect to an image signal supplied from thesurrounding area image capture unit 11.

The brightness adjuster 42, on the basis of a control signal from thecontrol processor 35, lowers the brightness of the non-monitor imagearea on the display unit 50 to less than the monitor image area. If thedisplay unit 50 is configured using display elements that may requireillumination, such as liquid crystal elements, for example, thebrightness adjuster 42 controls the illumination, such as the backlight,for example, to lower the brightness of the non-monitor image area toless than the monitor image area. Meanwhile, if the display unit 50 isconfigured using display elements that do not require illumination orusing light-emitting elements, such as organic EL display elements, forexample, a process may be conducted to lower the signal level of a lumasignal corresponding to the non-monitor image area.

According to the above configuration, the detection of a state ofstarting to view a display unit is conducted rapidly as a sequence ofdetecting a driver operation predicted in advance.

The steps of an ergonomic operation by an ordinary driver areanticipated as follows. When an animal grasps the outside world overallvia visual perception, eye movement grasps the direction and changesorientation the most rapidly, and when there is a desired object to viewin a range that is outside the rotation range of eye movement alone, theposture of the neck and head additionally changes to track the objectand compensate for the insufficient range.

As a result, in the series of steps by which the driver looks ahead andfocuses his or her line of sight in the installation direction of themirror unit which is installed in a significantly different direction,the driver grasps the approximate direction in which the image in themirror unit is visible, and starts eye rotation and neck rotation. Afterthe driver brings the mirror unit into the line of sight, as the neckturns, the eye rotation aligns with the line of sight direction so as tocancel out the neck rotation, and at the time when the line of sightbecomes aligned with the mirror unit, the neck-turning movement alsostops and becomes approximately still. The head rotation state that ischaracteristic of this series of operations is continuously analyzedover time to learn the change from the state of motion acceleration anddeceleration to the state of starting to view the mirror unit. Byconducting learning in this way and analyzing the features of theoperation sequences of individual drivers, it becomes possible todetermine the start of a driver focusing on the mirror unit startingfrom before that driver's head rotation is perfectly still.Additionally, by rapidly proceeding to instruction operation next, alow-lag human-machine interface (HMI) for screen operations is realized.At this point, the determination trend of the head orientationrecognition may be provided to the driver DR as visual or audiblefeedback via a notification unit, such as an overlay display on thedisplay of the display unit 50, an LED display, or a speaker in thevehicle, for example. In so doing, it is possible to realizeconfirmation of an instruction operation more accurately and with theminimum level of operation. It is desirable for the feedback to thedriver DR to be not a digital 0/1, true or false determination, butrather analog feedback by the notification unit of a status indicatingthe degree of determination of the driver's motion, for example.However, since the display of too much feedback may also impede thevisual field, and if such a display becomes unwanted after learning thecharacteristics unique to the driver, the display function may also bestopped, and the display may not be displayed all the time.

In addition, when the driving is ordinary high-speed forward travel,there is little bodily motion and a narrow range becomes the targetrange of vision, whereas during low-speed parking, the driver makeslarge movements of the head and body as part of the work to grasp theentire surroundings. For this reason, it is desirable for thedetermination reference used to detect each instruction to differbetween high speed and low speed. Consequently, operability is improvedby having the mode change determination perform a correlationdetermination of the driving state of the vehicle.

4-2. Operation of Fourth Embodiment

FIG. 28 is a flowchart illustrating operation of a display controller.In step ST41, the display controller 20 acquires a determinationreference value. The display controller 20 acquires a determinationreference value to use for the determination of a glance operation, aninstruction to change the visible range, and an instruction to restorethe visible range. Also, a determination reference value may be providedfor each driver, and a determination reference value corresponding tothe current driver may also be acquired. In addition, the determinationreference value may be updated according to driver operations, and thelatest determination reference value may also be acquired. The displaycontroller 20 acquires a determination reference value, and proceeds tostep ST42.

In step ST42, the display controller 20 starts observation of the headorientation. The display controller 20 uses an image signal suppliedfrom the driver image capture unit 12 to conduct face recognition,starts a process of determining the driver's face and a process ofdetecting the orientation of the determined head, and proceeds to stepST43.

In step ST43, the display controller 20 determines the position andorientation of the face during steady-state. The display controller 20determines the position and orientation of the face during steady-stateon the basis of the observation result of the face orientation (headorientation). When the face position or orientation during steady-statechanges from the previous determination, the display controller 20computes a correction value according to the magnitude of the change.Also, if there is no information corresponding to the driver, thedifference from a default value is treated as the correction value. Thedisplay controller 20 computes a correction value and proceeds to stepST44.

In step ST44, the display controller 20 configures a glancedetermination reference value. The display controller 20 uses adetermination reference value and the computed correction value toconfigure a glance determination reference value, and proceeds to stepST45.

In step ST45, the display controller 20 starts a glance determination.The display controller 20 starts a process of determining a glanceoperation of the driver using an observation result of the headorientation and the glance determination reference value, and proceedsto step ST46.

In step ST46, the display controller 20 tracks the face position andorientation. The display controller 20 tracks the face position andorientation on the basis of an observation result of the face positionand orientation, and proceeds to step ST47.

In step ST47, the display controller 20 determines whether or not achange has occurred in the face position or orientation. The displaycontroller 20 returns to step ST46 in the case of determining no changein the face position or orientation, and proceeds to step ST48 in thecase of determining that a change has occurred.

In step ST48, the display controller 20 determines whether or not amirror unit gaze state exists. The display controller 20 uses the faceposition and orientation and a determination reference value todetermine whether or not the driver is in a state of gazing at themirror unit. The display controller 20 proceeds to step ST49 in the caseof determining a gaze state, and returns to step ST46 in the case ofdetermining no gaze state. Note that the gaze state herein refers to apoint at which there is a moment of the eyes focusing and proceeding tograsp the situation as a physiological operation, and is not limited tobeing a state in which the driver is intently watching an image in themirror unit.

In step ST49, the display controller 20 switches to a high-speeddetection mode. The display controller 20 conducts high-speedobservation of the face position and orientation in order to preciselydetect an instruction to change the visible range and an instruction torestore the visible range by the driver, thereby enabling fine motion ofthe driver to be detected. In the case of high-speed detection mode, thedisplay controller 20 periodically conducts observation of the faceposition and orientation of the driver at a frequency of 100 ms or less,for example. Note that glance detection is conducted in an ordinarydetection mode, in which observation of the face position andorientation of the driver is periodically conducted over a widerinterval than the high-speed detection mode. The display controller 20switches to high-speed detection mode, and proceeds to step ST50.

In step ST50, the display controller 20 starts instruction detection.The display controller 20 starts the detection of an instruction tochange the visible range and an instruction to restore the visible rangeusing an observation result of the face position and orientation as wellas a determination reference value, and proceeds to step ST51.

In step ST51, the display controller 20 determines whether or not aninstruction is detected. If driver motion indicating an instruction tochange the visible range or an instruction to restore the visible rangeis detected by instruction detection, the display controller 20 proceedsto step ST52. Meanwhile, if an instruction is not detected, the displaycontroller 20 proceeds to step ST53.

In step ST52, the display controller 20 controls the display accordingto an instruction. If an instruction to change the visible range isdetected, for example, the display controller 20 changes the visiblerange displayed by an image in the monitor image area according to achange in the face position and orientation. In addition, if aninstruction to restore the visible range is detected, for example, thedisplay controller 20 reverts the visible range displayed by an image inthe monitor image area to the area before being changed. The displaycontroller 20 controls the display in this way, and returns to stepST50.

In step ST53, the display controller 20 determines whether or not thehead orientation is in a direction different from the mirror unitdirection. The display controller 20 uses an observation result of thehead orientation and a determination reference value to determinewhether or not the head orientation is in a direction different from animage viewing direction. If the head orientation is determined to be ina direction different from the image viewing direction, the displaycontroller 20 switches from high-speed detection mode to ordinarydetection mode, and returns to step ST47. Meanwhile, if the headorientation is determined to be in the direction of the mirror unit, thedisplay controller 20 proceeds to step ST54.

In step ST54, the display controller 20 determines whether or not thetime is within a predetermined determination period. If the time iswithin the predetermined determination period, the display controller 20returns to step ST50, whereas if the predetermined determination periodhas elapsed, the display controller 20 switches the high-speed detectionmode to ordinary detection mode, and returns to step ST47.

FIG. 29 is a diagram illustrating an example of the operation of aglance and an instruction to change the visible range. Also, FIG. 30 isa diagram illustrating an example of the operation of glancedetermination and determination of an instruction to change the visiblerange. Note that FIG. 30(A) illustrates an example of the relationshipbetween the glance direction (angle) and time, while FIG. 30(B)illustrates an example of the relationship between the glance speed andtime.

In FIG. 29, the direction PF1 indicates the front direction that thedriver's head faces normally, while the direction PF2 indicates anexample of the head orientation when the driver views an image in themirror unit 55. The direction PF3 indicates the direction of the mirrorunit 55.

The range FS1 indicates an example of the movement range of the driver'shead orientation normally, while the range FSa indicates an example ofthe search range by the driver's eye movement. The range FS2 is a headrotation acceleration zone when the head orientation moves from thedirection PF1 to the direction PF2, and a head rotation decelerationzone when the head orientation moves from the direction PF2 to thedirection PF1. The range FS3 is a head rotation deceleration zone whenthe head orientation moves from the direction PF1 to the direction PF2,and a head rotation acceleration zone when the head orientation movesfrom the direction PF2 to the direction PF1. The angle FVa illustratesan example of the glance angle of the head when the driver views animage in the mirror unit 55. The angle FVb illustrates an example of theglance angle covered by eye movement when the driver views an image inthe mirror unit 55. The range FSc illustrates an example of the searchrange by eye movement when the driver views an image in the mirror unit55.

In FIG. 30(A), the range JA is the range of the glance angle interpretedas the driver facing ahead. The range JB is the range of the glanceangle interpreted as the driver gazing at the mirror unit 55 when thedriver rotates his or her head in the direction PF2. The range JC is therange of the glance angle interpreted as gazing at the mirror unit 55 byeye movement. The curve QCa indicates the head orientation, while thecurve QCb indicates a line of sight direction based on the headorientation and eye movement. Also, in FIG. 30(B), the range JE is thedetermination range during a forward glance.

The display controller 20 detects that the glance speed (correspondingto the change in the head orientation) exceeded the range JE indicatedby the determination reference value and returned to within the rangeJE, while the glance direction (angle) is within the range JB (a rangeconfigured with reference to the mirror direction) of the glancedirection indicated by the determination reference value. In this case,the display controller 20 estimates that the driver has glanced and isgazing in the direction of the mirror unit 55, and determines that aglance operation is conducted. Note that if eye movement detection isconducted and the line of sight direction is estimated on the basis ofthe head orientation and the eye movement, a glance operation may bedetermined more precisely by comparing the estimated line of sightdirection to the range JC. In the case of determining that a glanceoperation is conducted, the display controller 20 changes to thehigh-speed detection mode to enable the detection of an instruction tochange the visible range with a high time resolution.

Note that in a rearview monitoring system on the driver side and thepassenger side, it is anticipated that the position and direction of thedisplay unit 50 and the mirror unit 55 may differ, and instructionoperation determination, glance angle determination, and the like mayall differ. For this reason, it is desirable to configure the bases ofdetermination individually.

The display controller 20 compares the range JB for determining whetheror not the glance direction (angle) is in a gaze state to the glancedirection (angle), and detects that the glance direction (angle)exceeded the range JB in a predetermined direction a predeterminednumber of times, such as two times, for example. Herein, in the case ofdetecting a combination of two or more acceleration/decelerationoperations of the driver's head at the time tj as illustrated in FIG.30(A), or in other words, in the case of determining that the glancedirection (angle) exceeded the range JB two times, the displaycontroller 20 estimates that the driver is performing an instruction inthe gaze state with respect to the mirror unit, and determines that aninstruction to change the visible range is conducted. After that, thedisplay controller 20 controls the display to change the visible rangeaccording to the instruction to change the visible range.

Note that the display controller 20 detects that the head orientation iswithin the range JA indicated by the determination reference value. Inthis case, the display controller 20 determines that the driver changedorientation from the mirror unit direction to the forward direction, andreturns to the state before the glance operation determination.

In addition, the operating instruction magnitude used to perform theinstruction to change the visible range may also differ depending on thedriving state. Typically, the driver is used to making large movementswith his or her head and body during low-speed driving operations suchas parking. On the other hand, during high-speed travel, there is littlemovement of the head primarily, and eye movement dominates the majorline of sight movement. For this reason, it is preferable to provide abasis of determination with less head movement. Furthermore, althoughFIG. 30 illustrates an example of two or more repeated movements, thedriver's head operation may also be a series of one-way, two-stageacceleration motions with no substantial reverse motion.

FIG. 31 illustrates an example of a case of using the acceleration of ahead rotation operation as an operating instruction magnitude. FIG.31(A) illustrates the glance direction (angle), while FIG. 31(B)illustrates the glance speed, and FIG. 31(C) illustrates the glanceacceleration.

When a driver performs a glance multiple times, the reverse operation ofthe glance and the reverse operation of the next glance may not bestrictly aligned at the same start position. For example, the nextglance operation may be conducted before the reverse operation of theprevious glance returns to the original position. In this case, thecurve QCa indicating the head orientation varies as illustrated in FIG.31(A). Consequently, determining an instruction to change the visiblerange on the basis of the glance direction (angle) may be difficult.However, if the reverse operation of the glance is repeated, the glanceacceleration becomes a waveform producing one cycle of oscillation perglance operation, as illustrated in FIG. 31(C). Consequently, thedisplay controller 20 may utilize the glance acceleration to detect therepetition of a glance precisely.

Furthermore, the display controller 20 learns an operation featurehistory of a driver's glance speed and glance direction, and changes thedetermination reference value for each driver according to the featuresunique to that driver. By updating the determination reference value inthis way, a self-learning function is provided that improves theprecision of determining a glance operation or an instruction to changethe visible range, and additionally enables accurate instructiondetection with fewer instruction operations. Also, with the learningfunction, operation features, including the operating instructionmagnitude, are self-learned to produce work optimized for the featuresof the driver.

By conducting such processing, it is possible to conduct precise displaycontrol corresponding to operations performed when the driver grasps theconditions of the surrounding area.

In addition, the detection of the movement of the head orientation isconducted after a time for the glance operation to stabilize haselapsed. In so doing, when head movement in a direction orthogonal tothe head orientation is repeated two or more times, or when aface-turning operation that rotates the orientation of the face aboutthe axis of the neck is repeated two or more times, for example, it isdetermined that an instruction to change the visible range or aninstruction to restore the visible range has been conducted.Consequently, the determination of an instruction may be conducted evenmore precisely.

The present technology, on the basis of an ergonomic procedure when aperson glances in a different direction from a line of sight directionof primary attention, accounts for the motion to stabilize head movementas a target object is rapidly grasped by eye movement and the eyes startto focus while at the same time the person finally starts paying visualattention. In other words, in the moment at which the driver visuallyperceives the display content and starts to grasp the situation, themovement of the head position stabilizes and beings to stop. The driveradditionally judges, by perceiving the screen, whether or not to changethe visible range, thereby reflecting that an instruction operation isconducted after the stabilization procedure is experienced.

Also, in the display control, the display controller 20 controls thedisplay in response to detecting that the driver's head orientation hasmoved in the direction of the mirror unit. For example, if the headorientation is not in the direction of the mirror unit, the backlight isturned off or the signal level of the luma signal is lowered so that theimage is not visible or less visible. Meanwhile, if the head orientationis in the direction of the mirror unit, the backlight is turned on andthe brightness control or the signal level of the luma signal isadjusted to enable the driver to check an image of the visible range.Consequently, when the driver is facing ahead, the display of aneedlessly bright image of the surrounding area may be avoided. Inaddition, an image of the surrounding area may be displayed only whenthe driver is facing in the direction of the mirror unit.

In addition, after changing the visible range, the display controller 20may also gradually and automatically revert the visible range back tothe original position. In this case, the driver is not required toperform an operation to revert the visible range back to the originalrange. Furthermore, in order to avoid a state in which the drivermomentarily becomes unable to check the situation, the displaycontroller 20 controls the display so that changes in the display scaleand changes in the display format do not occur suddenly.

For the sake of convenience, the description in this specificationdescribes a practical example of a procedure of recognizing the headdirection and line of sight, and detecting an instruction. However,since the actual aim is not to compute a specific head direction andline of sight accurately, it is sufficient to obtain a correlation withthe state of the driver viewing the display unit 50, and the role of thedisplay controller 20 does not necessarily require the provision of afunction enabling line of sight recognition or an accurate grasping ofthe head orientation. For this reason, the display controller 20 is notlimited to the configuration of FIG. 27, and may be configureddifferently insofar as the display is responsive to the vehicle stateand operating instruction gestures by the driver.

5. Other Embodiments

In addition, as another embodiment, when displaying an image on thedisplay unit 50, if the image is compressed for display, there is a riskthat the apparent distance perceived from the display image may differgreatly from the distance to a subject included in the surrounding area.Accordingly, a warning display may also be provided on the screen or ona border of the display unit, according to the degree of imagecompression.

FIG. 32 illustrates an example of a configuration of a displaycontroller in the case of presenting a warning display. The displaycontroller 20 is equipped with a driver movement determination unit 21,a control processor 35, a display adjuster 41, a brightness adjuster 42,and a warning display overlay unit 43.

The driver movement determination unit 21, on the basis of an imagesignal supplied from the driver image capture unit 12, detects the headposition of the driver, and determines the movement direction andmovement magnitude of the head position of the driver. For example, thedriver movement determination unit 21 conducts facial recognition of thedriver on the basis of an image signal supplied from the driver imagecapture unit 12, for example, and determines the position of therecognized face, the orientation of the face or the orientation of thehead. Furthermore, the driver movement determination unit 21 tracks therecognized face, and determines the movement direction and movementmagnitude of the head position. The driver movement determination unit21 outputs a determination result to the control processor 35.

The control processor 35, on the basis of a determination result fromthe driver movement determination unit 21, generates a control signal tocontrol the display differently for the monitor image area and thenon-monitor image area of the display unit 50, and outputs to thedisplay adjuster 41, the brightness adjuster 42, and the warning displayoverlay unit 43.

The display adjuster 41, on the basis of a control signal from thecontrol processor 35, conducts a scale adjustment of the captured imageof the surrounding area, changes or combines images of the surroundingarea, and the like with respect to an image signal supplied from thesurrounding area image capture unit 11.

The brightness adjuster 42, on the basis of a control signal from thecontrol processor 35, lowers the brightness of the non-monitor imagearea on the display unit 50 to less than the monitor image area. If thedisplay unit 50 is configured using display elements that may requireillumination, such as liquid crystal elements, for example, thebrightness adjuster 42 controls the illumination, such as the backlight,for example, to lower the brightness of the non-monitor image area toless than the monitor image area. Meanwhile, if the display unit 50 isconfigured using display elements that do not require illumination orusing light-emitting elements, such as organic EL display elements, forexample, a process may be conducted to lower the signal level of a lumasignal corresponding to the non-monitor image area.

The warning display overlay unit 43, on the basis of a control signalfrom the control processor 35, causes information indicating the degreeof image compression, for example, to be overlaid onto an image afterdisplay-variable processing. For example, an image of the visible rangeis displayed to match left and right motion of the face, the visiblerange is enlarged by a repeated operation of the head or face two ormore times, and the display before enlargement is restored by anoperation in the opposite direction. In addition, when displaying animage with a changed scale, a dynamic warning display is provided towarn that the display has a changed scale. To intuitively understand thewarning content, the dynamic warning display adjusts a frame sizeaccording to the operation of changing the scale, displaying azebra-striped frame in which the zebra stripes flow according to thechange of scale. Regarding the flow of the zebra display, when takinghuman physiological mechanisms into account, it is desirable to includemotion in an approaching direction, since humans are sensitive toapproaching objects, which are perceived as a threat when a movingobject enters the visual field from outside the line of sight.

By controlling the display in this way, when an image is compressed fordisplay, a loss of the driver's sense of distance and the like may beavoided. For example, it is possible to avoid the misrecognition of anapproaching subject as a subject at a distant position due to the imagebeing compressed.

In addition, the display controller 20 may also control the display inconjunction with a navigation system, and display an image with a widerangle of view than the ordinary rearview visual field when merging on anexpressway or traveling in a roundabout.

In addition, the determination of driver motion is not limited to usingmovement of the head orientation, or in other words, head rotation aboutthe axis of the neck, and the display may also be controlled by using atleast one of a head tilt operation (nodding) or movement in the front,back, left, or right direction of the driver's upper body, for example.Furthermore, an operation of moving the head or a posture change in thedirection of the mirror unit may also be used to conduct a zoom-in orzoom-out operation on the display image.

FIGS. 33 to 35 illustrate examples of display control based on otherdriver motions. FIG. 33 illustrates an example of the placement of asurrounding area image capture unit and a surrounding image displayed ona display unit. As illustrated in FIG. 33(A), surrounding area imagecapture units 11 c and 11 d are provided on the sides of the vehicle,while a surrounding area image capture unit 11 e is provided on the rearface of the vehicle. As illustrated in FIG. 33(B), the displaycontroller 20 displays an image of three surrounding areas acquired bythe surrounding area image capture units 11 c, 11 d, and 11 e on thedisplay unit 50.

FIG. 34 is a diagram for explaining the changing of a surrounding areaimage when the head moves in a front-back direction. As illustrated inFIG. 34(A), before the head movement is conducted, the displaycontroller displays an image of three surrounding areas acquired by thesurrounding area image capture units 11 c, 11 d, and 11 e on the displayunit 50. In addition, in the case of determining head movement in theforward direction, for example, as illustrated in FIG. 34(B), thedisplay controller changes to an image in which the display areasdisplaying images of the surrounding area acquired by the surroundingarea image capture units 11 c and 11 d provided on the vehicle sides areextended in the advancing direction. After that, when the head movesback to the original position, the display controller controls thedisplay to revert to the image before the change.

FIG. 35 is a diagram for explaining the changing of a surrounding areaimage when the head moves in a leftward direction. As illustrated inFIG. 35(A), before the head movement is conducted, the displaycontroller displays an image of three surrounding areas acquired by thesurrounding area image capture units 11 c, 11 d, and 11 e on the displayunit 50. In addition, in the case of determining head movement in theleftward direction, for example, as illustrated in FIG. 35(B), thedisplay controller changes to an image with an extended display areadisplaying an image of the surrounding area on the left side. Afterthat, when the head moves back to the original position, the displaycontroller controls the display to revert to the image before thechange.

In this way, by controlling the display by using not only movement ofthe head orientation but also factors such as head tilt operations,images of the surrounding area may be displayed in a variety of formats.

Furthermore, although the foregoing embodiment describes the case ofchanging the visible range by controlling the display on the basis of adriver motion determination result, the orientation of the mirror unitmay also be changed on the basis of a driver motion determinationresult, thereby depicting, in the mirror unit, an image of a desiredvisible range of the display image on the display unit.

In addition, as a configuration using a semi-transmissive mirror, themirror unit may also be provided with a display device behind the mirrorunit, such as liquid crystal display elements, organic EL displayelements, or display elements using light-emitting diodes, for example.In this case, in the mirror unit, an image of the surrounding areadisplayed on the display unit may be viewed at the same time as an imagedisplayed on the back display device. Consequently, if variousinformation is displayed on the back display device, the driver would beable to read information while also checking the surrounding area at thesame time. The information to displayed with the back display device maybe information related to an image of the surrounding area displayed onthe display unit (such as a determined instruction from the driver or adetermined mode of the visible range, for example), or informationrelated to the driving conditions or the like (such as navigationinstruction information).

Furthermore, the mirror unit may be configured so that thesemi-transmissive mirror includes a semi-transmissive light dimmingfunction, or so that a light dimming device that adjusts the amount oftransmitted light is additionally disposed behind the semi-transmissivemirror. According to such a configuration, it becomes possible to usethe semi-transmissive light dimming function or the light dimming deviceto adjust the visibility of a display image on the display deviceprovided behind the mirror unit. Consequently, information may bedisplayed so that the driver is able to read various information easilywhile also checking the surrounding area.

In addition, the foregoing embodiment illustrates an example of a casethat realizes a function corresponding to a rearview mirror. However,since the viewing of the surrounding area is conducted with not onlyouter rearview mirrors but also an inner rearview mirror, a functioncorresponding to an inner rearview mirror may also be realized. Next,the case of realizing a function corresponding to an inner rearviewmirror will be described as another embodiment. Note that in this case,the display unit, mirror unit, and the like are not strictly limited toan arrangement corresponding to mirrors of the past, and may also bearranged at positions such as near the instrument panel or console panelon the dashboard, for example.

First, the case of using a single surrounding area image capture unit torealize a function corresponding to an inner rearview mirror will bedescribed. FIG. 36 illustrates an example of a configuration in the caseof using a single surrounding area image capture unit to realize afunction corresponding to an inner rearview mirror. For example, asillustrated in FIG. 36(A), a single surrounding area image capture unitis provided at the rear of the vehicle to capture an image behind thevehicle, enabling the driver DR to check an image of the surroundingarea acquired by the surrounding area image capture unit 11. Note thatin FIG. 36(A), the display unit that displays the image of thesurrounding area has been omitted.

In addition, even if the driver DR views a surrounding area acquired bythe surrounding area image capture unit 11, the driver DR is able tomaintain a sense of distance similar to a surrounding area reflected inan inner rearview mirror 61. For example, the area ARu acquired by thesurrounding area image capture unit 11 (the horizontal angle of view ofa surrounding image acquired by the surrounding area image capture unit11) is approximately equal to the horizontal visual field area ARrm of asurrounding area reflected in the inner rearview mirror 61. In so doing,a single surrounding area image capture unit may be used to realize afunction corresponding to an inner rearview mirror.

Meanwhile, if a following car OBc or person OBp is positioned asillustrated in FIG. 36(A), the surrounding image reflected in the innerrearview mirror 61 is an image of the visual field area ARrm in FIG.36(A), and thus becomes the image illustrated in FIG. 36(B), forexample. In addition, the surrounding area captured by the surroundingarea image capture unit 11 is the surrounding area ARu in FIG. 36(A),and thus becomes the image illustrated in FIG. 36(C), for example. Inother words, in an image of the surrounding area captured by thesurrounding area image capture unit 11, the area of the blind spot nearthe vehicle is large compared to the surrounding image reflected in theinner rearview mirror 61, and thus the driver may be unable to check theperson OBp as illustrated in FIG. 36(C).

Accordingly, as illustrated in FIG. 37, a plurality of surrounding areaimage capture units are used to realize a function corresponding to aninner rearview mirror. For example, if a surrounding area image captureunit 11 g is provided on the left side of the vehicle while asurrounding area image capture unit 11 h is provided on the right sideof the vehicle, and each surrounding area image capture unit 11 capturesan image of the rear to realize a function corresponding to an innerrearview mirror, the blind spot is reduced compared to the case of FIG.36. Note that the area ARrm is the horizontal visual field area of thesurrounding image reflected in the inner rearview mirror 61, while thearea ARug is the horizontal visual field area of the surrounding imageacquired by the surrounding area image capture unit 11 g on the leftside, and the area ARuh is the horizontal visual field area of thesurrounding image acquired by the surrounding area image capture unit 11h on the right side. In so doing, when a surrounding area image captureunit is used to realize a function corresponding to an inner rearviewmirror, by combining images of the surrounding area acquired by thesurrounding area image capture unit 11 g and the surrounding area imagecapture unit 11 h, it is possible to avoid a larger blind spot area nearthe vehicle.

In addition, since parallax occurs between the images of the surroundingarea acquired by the surrounding area image capture unit 11 g and thesurrounding area image capture unit 11 h, it is difficult to cancel theparallax differences to combine and display the surrounding images.Consequently, for the combining of images of the surrounding area, animage combining process is made to be ergonomically and easily conductedso that parallax differences are not noticeable.

FIG. 38 illustrates an example of a configuration in the case ofreducing the blind spot compared to an inner rearview mirror. Asurrounding area image capture unit 11 j is provided on the left side ofthe vehicle, while a surrounding area image capture unit 11 k isprovided on the right side of the vehicle. In the surrounding area imagecapture unit 11 j, the horizontal visual field area (angle of view) ofthe acquired surrounding image is configured to include the area ARuj1included in the horizontal visual field area ARrm of the surroundingimage reflected in the inner rearview mirror 61, and the area ARuj2 onthe outward left side of the visual field area ARrm. In the surroundingarea image capture unit 11 k, the horizontal visual field area (angle ofview) of the acquired surrounding image is configured to include thearea ARuk1 included in the horizontal visual field area ARrm of thesurrounding image reflected in the inner rearview mirror 61, and thearea ARuk2 on the outward right side of the visual field area ARrm. Inaddition, regarding the rear position of the vehicle, the visual fieldarea ARrm is conducted to be included in at least one of the area ARuj1and the area ARuk1.

The display adjuster 41 of the display controller 20 discussed earlier,on the basis of a control signal from the control processor 35, trimsand joins images from the surrounding images acquired by the surroundingarea image capture units 11 j and 11 k. The control processor 35controls the image trimming so that the surrounding area is continuousin the joined image. Furthermore, the control processor 35 moves theimage trimming position and joining position according to an instructionfrom the driver, on the basis of a determination result by the drivermovement determination unit 21.

The control processor 35, by joining an image of the area ARmj from thesurrounding image acquired by the surrounding area image capture unit 11j and the area ARmk from the surrounding image acquired by thesurrounding area image capture unit 11 k, for example, enables thedriver to check a surrounding area that includes the area ARrm, asillustrated in FIG. 38(B). In addition, since an image used to check thesurrounding area may be generated by simply joining the image of thearea ARmj and the image of the area ARmk, without conducting a processto cancel parallax differences to combine and display images, an imagefor checking the surrounding area may be generated easily.

In addition, for the image joining, a blend process is conducted over arange predetermined with reference to the image joining position, andthe blend ratio of the surrounding image acquired by the surroundingarea image capture unit 11 j and the surrounding image acquired by thesurrounding area image capture unit 11 k is continuously varied. Byconducting such a process, the joined portion may become lessnoticeable.

In addition, when the image trimming position is varied on the basis ofa determination result from the driver movement determination unit 21,and the position of the driver DR moves to the left or right, forexample, the control processor 35 moves the area of the image to trimsimilarly to the movement of the surrounding area reflected in the innerrearview mirror 61. In so doing, a function corresponding to an innerrearview mirror may be realized, even if driver motion occurs.

Meanwhile, for the joining of the surrounding image acquired by thesurrounding area image capture unit 11 j and the surrounding imageacquired by the surrounding area image capture unit 11 k, since an imageof a right viewpoint and an image of a left viewpoint are joined, when arear vehicle or person is at the joining position, there is a risk thatthe image of the vehicle, person, or the like may become anunnatural-looking image. Particularly, if the vehicle, person, or thelike is approaching, the unnaturalness of the image becomes significantat the joined portion because the parallax is large. Accordingly, thecontrol processor 35 is able to change the joining position according toan instruction from the driver, on the basis of a determination resultfrom the driver movement determination unit 21.

FIG. 39 is a diagram for explaining the changing of an image joiningposition. As illustrated in FIG. 39(A), when the joining position of asurrounding image acquired by the surrounding area image capture unit 11j and a surrounding image acquired by the surrounding area image captureunit 11 k is the position Pb1, since the position Pb1 is not included inthe image area of a following car OBc or person OBp, the following carOBc or person OBp may be checked easily with the display surroundingimage. However, if the joining position is fixed at the position Pb1,and the image area of the person OBp becomes the position Pb1 asillustrated in FIG. 39(B), the image of the person OBp becomes a joinedimage of a left viewpoint image and a right viewpoint image, and thereis a risk that the person OBp may become difficult to correctlydistinguish. Accordingly, the control processor 35 changes the joiningposition according to an instruction from the driver. For example, inthe case of determining a driver instruction to move the joiningposition to the left, the joining position moves to the left, asillustrated in FIG. 39(C). If the joining position is moved in this way,the person OBp is displayed by only the surrounding image acquired bythe surrounding area image capture unit 11 k, and thus it becomespossible to easily and correctly distinguish the person OBp.

In addition, if the control processor 35 is configured to present adisplay indicating the joining position (for example, a display such asa marker), the driver is able to easily grasp the positionalrelationship between the joining position and a following car, person,or the like. For this reason, the driver becomes able to easilydetermine whether or not to move the joining position, and determine apreferable direction in which to move the joining position. Note thatthe default position of the joining position may be automatically set toa preconfigured position or a user-configured position, and may also bethe position that was configured when the previous drive ended.

In addition, the instruction from the driver is detected from anoperating instruction gesture performed by the driver using ahuman-machine interface responsive to the driver operations discussedabove, for example. In so doing, the driver becomes able to easily movethe joining position in a desired direction by simply moving his or herhead or line of sight.

Furthermore, the processing sequence that is explained in thespecification can be implemented by hardware, by software and by aconfiguration that combines hardware and software. In a case where theprocessing is implemented by software, it is possible to install inmemory within a computer that is incorporated into dedicated hardware aprogram in which the processing sequence is encoded and to execute theprogram. It is also possible to install a program in a general-purposecomputer that is capable of performing various types of processing andto execute the program.

For example, the program can be recorded on a hard disk, SSD (SolidState Drive), or read-only memory (ROM) that is built into a signalprocessing semiconductor device or provided separately as a recordingmedium in advance. Alternatively, the program can be temporarily orpermanently stored (recorded) in (on) a removable recording medium suchas a flexible disk, CD-ROM (Compact Disc Read Only Memory), MO (MagnetoOptical) disk, DVD (Digital Versatile Disc), BD (Blu-Ray Disc(registered trademark)), a magnetic disk, or a semiconductor memorycard. Such a removable recording medium can be provided as so-calledpackage software.

In addition, the program can be, not only installed on a computer from aremovable recording medium, but also transferred wirelessly or by wire,via a self-diagnostic function or on-board diagnostics (OBD) terminal,to the computer from a download site via a network such as a LAN (LocalArea Network) or the Internet. In such a computer, a program transferredin the aforementioned manner can be received and installed or updated ona recording medium such as built-in hardware.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

Additionally, the vehicle display device according to an embodiment ofthe present technology may also be configured as below.

(1)A vehicle display device including:a display unit, installed in a cabin of a vehicle and having a displayface oriented in a different direction from a direction of a driver,configured to display an image depicting a surrounding area of thevehicle; anda mirror unit installed in the cabin of the vehicle and configured toreflect some or all of an image area displayed on the display unit,wherein a visible range of the surrounding area visible to the driverthrough an image of the display unit reflected in the mirror unitchanges according to movement of a viewing position of the driver withrespect to the mirror unit.(2)The vehicle display device according to (1), further including:a display controller configured to determine motion of the driver,control a display of an image to display on the display unit on thebasis of a motion determination result, and change the visible rangeaccording to the motion of the driver.(3)The vehicle display device according to (2), further including:a notification unit configured to present a change of determination ofthe motion of the driver to the driver as visual or audible feedback.(4)The vehicle display device according to (2), whereinthe display controller lowers a brightness of some or all of anotherdisplay area excluding a display area corresponding to the visiblerange, controls a range of a display area to be lowered in brightness onthe basis of a motion determination result, and changes the visiblerange according to the motion of the driver.(5)The vehicle display device according to (4), whereinthe display controller controls a range of a display area to be loweredin brightness on the basis of the motion determination result, andextends the visible range.(6)The vehicle display device according to any one of (2) to (5), whereinthe display controller conducts image compression in a movementdirection of the visible range with respect to an edge area in themovement direction of the visible range in an image depicting asurrounding area of the vehicle, controls a compression ratio or acompression range on the basis of a motion determination result, andchanges the visible range according to the motion of the driver.(7)The vehicle display device according to (6), whereinthe display controller controls a compression ratio or a compressionrange on the basis of a motion determination result, and extends thevisible range.(8)The vehicle display device according to any one of (2) to (7), whereinthe display controller conducts the image display on the display unit ina case of determining on the basis of a motion determination result thatthe driver glanced in a direction of the mirror unit.(9)The vehicle display device according to (8), whereinthe display controller extends an area of the visible range in a case ofdetermining that the driver performed a predefined motion afterdetection of the glance.(10)The vehicle display device according to (8) or (9), whereinthe display controller raises a time resolution for determination of themotion of the driver after the glance is detected.(11)The vehicle display device according to any one of (2) to (10), whereinthe display controller includesa learning unit configured to learn features of motion for the driverindividually,a determination processor configured to use a learning result of thelearning unit to determine the motion of the driver, anda control processor configured to change the visible range on the basisof a determination result of the determination processor.(12)The vehicle display device according to (11), whereinthe determination processor conducts a determination of an instructionby the driver from the motion of the driver, and in determination of theinstruction, determines content of the instruction on the basis of adetection result of a combination of two or more acceleration anddeceleration operations of the driver's head.(13)The vehicle display device according to (11) or (12), whereinthe motion of the driver is at least one of neck-turning, head rotationabout a neck axis, or upper body movement forward, back, left, or right.(14)The vehicle display device according to any one of (2) to (13), whereinthe display controller extends the visible range according to the motionof the driver, and controls an area size of the visible range to extendon the basis of driving conditions of the vehicle.(15)The vehicle display device according to (14), whereinthe display controller extends the visible range in an outward directionof the vehicle on the basis of driving information related to a traveldirection of the vehicle.(16)The vehicle display device according to (14) or (15), whereinthe vehicle includes a trailer part coupled to the cabin, and imagecapture of a surrounding area of the vehicle is conducted from thecabin, andthe display controller, on the basis of driving information related todriving conditions producing a situation of the cabin orientationdiffering from the trailer part orientation, extends the visible rangein a manner that an outward side of the trailer part is still includedin the visible range even if the cabin orientation and the trailer partorientation are different.(17)The vehicle display device according to any one of (1) to (16), whereinthe display unit and the mirror unit are arranged in a manner that partof an image displayed on the display unit is reflected in the mirrorunit, and an image area reflected in the mirror unit moves according tomovement of a viewing position of the driver with respect to the mirrorunit.(18)The vehicle display device according to any one of (1) to (17), whereinthe display unit and the mirror unit are arranged or optically designedin a manner that an optical viewing distance from a position of thedriver's eyes to a display face of the display unit via the mirror unitis at least 1.1 meters or greater.(19)The vehicle display device according to any one of (1) to (18), whereinthe display unit is provided in a manner that a display face orilluminating light of the display face on the display unit is notvisible to the driver.(20)The vehicle display device according to any one of (1) to (19), whereina shape of the mirror unit in a movement direction of the visible rangeis a curved shape projecting out towards the driver.(21)The vehicle display device according to (20), whereinthe mirror unit is shaped with a smaller curve in a central part than atan end in a movement direction of the visible range.(22)The vehicle display device according to any one of (1) to (21), furtherincluding:a display controller configured to determine motion of the driver andchange an orientation of the mirror unit on the basis of a motiondetermination result, and thereby change the visible range according tothe motion of the driver.(23)The vehicle display device according to any one of (1) to (22), whereinthe mirror unit is a semi-transmissive mirror.(24)The vehicle display device according to (23), whereinthe semi-transmissive mirror includes a semi-transmissive light dimmingfunction, or a light dimming device that adjusts an amount oftransmitted light is additionally disposed behind the semi-transmissivemirror.(25)A display system in vehicle, comprising:

-   -   a display screen configured to display an image corresponding to        a surrounding area of the vehicle; and    -   a surface configured to reflect at least a portion of the image        displayed by the display screen to a viewer.        (26)        The display system according to (25), wherein the viewer is an        operator of the vehicle.        (27)        The display system according to (26), further comprising:        a control circuit configured to control display of the image of        the surrounding area based on a detection result of a state of        the viewer.        (28)        The display system according to (27), wherein,        the detection result includes a detection result of a motion of        the user.        (29)        The display system according to any one of (25) to (28), wherein        the surface is non-planar.        (30)        The display system according to (27), wherein the control        circuit is further configured to control display of the image of        the surrounding area based on a status of the vehicle.        (31)        The display system according to (30), wherein when a trailer is        attached to the vehicle, the status of the vehicle includes        whether the trailer blocks a portion of the surrounding area.        (32)        The display system according to (27), wherein the control        circuit is further configured to    -   perform, on the image, image compression in a movement direction        of a visible range of the surrounding area with respect to an        edge area in the movement direction of the visible range,    -   control at least one of a compression ratio and a compression        range based on the motion of the viewer, and    -   change the visible range according to the motion of the viewer.        (33)        The display system according to (32), wherein the control        circuit is further configured to control at least one of the        compression ratio and the compression range based on the motion        of the viewer in order to extend the visible range.        (34)        The display system according to (27), wherein the control        circuit is further configured to control image display on the        display screen when, based on the motion of the viewer, the        control circuit determines that the viewer looks in a direction        of the surface.        (35)        The display system according to (34), wherein the control        circuit is further configured to extend the portion of the        surrounding area that is displayed when the viewer performs a        predefined motion after looking in the direction of the surface.        (36)        The display system according to (34), wherein the control        circuit is further configured to raise a time resolution for        determining the motion of the viewer after determining that the        viewer looks in the direction of the surface.        (37)        The display system according to (27), wherein the control        circuit is further configured to    -   learn features of motion of the viewer,    -   use a learning result to determine the motion of the viewer, and    -   change the portion of the surrounding area that is displayed        based on a determination result of the motion of the viewer.        (38)        The display system according to (36), wherein the motion of the        viewer includes at least one of neck-turning, heat rotation        about a neck axis, or upper body movement forward, backward,        left or right.        (39)        The display system according to (27), wherein the control        circuit is further configured to    -   extend the portion of the surrounding area that is displayed        according to the motion of the viewer, and    -   control an area size of the portion of the surrounding area that        is displayed to extend based on conditions of the vehicle.        (40)        The display system according to claim (39), wherein the control        circuit is further configured to extend the portion of the        surrounding area that is displayed on an outward direction of        the vehicle based on information related to a travel direction        of the vehicle.        (41)        The display system according to (39), wherein the vehicle is a        vehicle that includes a cabin and a trailer coupled to the        cabin, and an image capture of the surrounding area is conducted        from the cabin, and    -   the display control circuit, based on driving information of the        vehicle, is further configured to extend the portion of the        surrounding area that is displayed to cause the trailer to be        displayed when the cabin orientation and the trailer orientation        are different.        (42)        The display system according to any one of (25) to (41), wherein        the display screen and the surface are arranged such that part        of the image displayed on the display screen is reflected on the        surface, and an image area reflected by the surface moves        according to movement of a viewing position of the viewer with        respect to the surface.        (43)        The display system according to any one of (25) to (43), wherein        the display screen and the surface are arranged or optically        designed to cause an optical distance from a position of the        viewer's eyes to a display face of the display screen via the        surface to be at least 1.1 meters or greater.        (44)        The display system according to any one of (25) to (43), wherein        the display screen is configured such that a display face or        illuminating light of the display face of the display screen is        not visible to the viewer.        (45)        The display system according to any one of (25) to (44), wherein        a shape of the surface in a movement direction of the portion of        the surrounding area that is displayed is a curved shape        projecting out towards the viewer.        (46)        The display system according to claim (45), wherein the surface        is shaped with a smaller curve in a central region than an end        region a movement direction of the viewer.        (47)        The display system according to any one of (25) to (46), further        comprising:    -   a control circuit configured to determine motion of the viewer        and change an orientation of the surface based on the motion of        the viewer determined, the portion of the surrounding area that        is displayed changing in accordance with the change in        orientation of the surface.        (48)        The display system according to any one of (25) to (47), wherein        the surface is a mirror.        (49)        The display system according to (49), wherein the mirror is a        semi-transmissive mirror.        (50)        The display system according to (49), wherein the        semi-transmissive mirror includes a semi-transmissive light        dimming function, or a light dimming device that adjusts an        amount of transmitted light, the light dimming device being        disposed behind the semi-transmissive mirror.        (51)        The display system according to any one of (25) to (50), further        comprising at least two or more image capture devices attached        to an exterior of the vehicle and facing rearward with respect        to a travel direction of the vehicle, wherein the image        corresponding to a surrounding area of the vehicle is captured        by at least one of the image capture devices.        (52)        The display system according to any one of (25) to (51), wherein        the vehicle is a car or a bus.        (53)        A display control method comprising:        displaying, with a display screen, an image corresponding to a        surrounding area of a vehicle; and    -   reflecting, with a surface, at least a portion of the image        displayed by the display screen to a viewer.

INDUSTRIAL APPLICABILITY

In a vehicle display device, display control method, and rearviewmonitoring system according to an embodiment of the present technology,an image depicting the area around a vehicle is displayed on a displayunit that is installed in the vehicle cabin and whose display face isoriented in a different direction from the direction of the driver. Amirror unit is installed in the cabin of the vehicle and reflects animage displayed on the display unit. The visible range of thesurrounding area that is visible to the driver through the image of thedisplay unit reflected in the mirror unit changes according to movementof the viewing position of the driver with respect to the mirror unit.For this reason, even in the case of enabling a surrounding area to bechecked by capturing and displaying an image of the area surrounding avehicle on a display unit, the area surrounding the vehicle may beviewed easily. The technology is suitable for a vehicle such as anautomobile, truck, or bus.

REFERENCE SIGNS LIST

-   -   10 display device    -   11, 11 a, 11 b, 11 surrounding area image capture unit    -   12 driver image capture unit    -   13 driving conditions detection sensor    -   15 driver identification information acquisition unit    -   20 display controller    -   21 driver movement determination unit    -   22 driving conditions determination unit    -   23 recognition unit    -   24 glance determination unit    -   25 instruction operation determination unit    -   26 driver authentication unit    -   35 control processor    -   41 display adjuster    -   42 brightness adjuster    -   43 warning display overlay unit    -   50 display unit    -   55 mirror unit    -   61 inner rearview mirror    -   91 rearview mirror    -   241 glance determination processor    -   242 glance determination learning unit    -   243 determination reference value storage unit    -   251 instruction operation determination processor    -   252 instruction operation determination learning unit    -   253 determination reference value storage unit

1. A display system in vehicle, comprising: a display screen configuredto display an image corresponding to a surrounding area of the vehicle;and a surface configured to reflect at least a portion of the imagedisplayed by the display screen to a viewer.
 2. The display systemaccording to claim 1, wherein the viewer is an operator of the vehicle.3. The display system according to claim 2, further comprising: acontrol circuit configured to control display of the image of thesurrounding area based on a detection result of a state of the viewer.4. The display system according to claim 3, wherein, the detectionresult includes a detection result of a motion of the user.
 5. Thedisplay system according to claim 1, wherein the surface is non-planar.6. The display system according to claim 3, wherein the control circuitis further configured to control display of the image of the surroundingarea based on a status of the vehicle.
 7. The display system accordingto claim 6, wherein when a trailer is attached to the vehicle, thestatus of the vehicle includes whether the trailer blocks a portion ofthe surrounding area.
 8. The display system according to claim 3,wherein the control circuit is further configured to perform, on theimage, image compression in a movement direction of a visible range ofthe surrounding area with respect to an edge area in the movementdirection of the visible range, control at least one of a compressionratio and a compression range based on the motion of the viewer, andchange the visible range according to the motion of the viewer.
 9. Thedisplay system according to claim 8, wherein the control circuit isfurther configured to control at least one of the compression ratio andthe compression range based on the motion of the viewer in order toextend the visible range.
 10. The display system according to claim 3,wherein the control circuit is further configured to control imagedisplay on the display screen when, based on the motion of the viewer,the control circuit determines that the viewer looks in a direction ofthe surface.
 11. The display system according to claim 10, wherein thecontrol circuit is further configured to extend the portion of thesurrounding area that is displayed when the viewer performs a predefinedmotion after looking in the direction of the surface.
 12. The displaysystem according to claim 10, wherein the control circuit is furtherconfigured to raise a time resolution for determining the motion of theviewer after determining that the viewer looks in the direction of thesurface.
 13. The display system according to claim 3, wherein thecontrol circuit is further configured to learn features of motion of theviewer, use a learning result to determine the motion of the viewer, andchange the portion of the surrounding area that is displayed based on adetermination result of the motion of the viewer.
 14. The display systemaccording to claim 13, wherein the motion of the viewer includes atleast one of neck-turning, heat rotation about a neck axis, or upperbody movement forward, backward, left or right.
 15. The display systemaccording to claim 3, wherein the control circuit is further configuredto extend the portion of the surrounding area that is displayedaccording to the motion of the viewer, and control an area size of theportion of the surrounding area that is displayed to extend based onconditions of the vehicle.
 16. The display system according to claim 15,wherein the control circuit is further configured to extend the portionof the surrounding area that is displayed on an outward direction of thevehicle based on information related to a travel direction of thevehicle.
 17. The display system according to claim 15, wherein thevehicle is a vehicle that includes a cabin and a trailer coupled to thecabin, and an image capture of the surrounding area is conducted fromthe cabin, and the display control circuit, based on driving informationof the vehicle, is further configured to extend the portion of thesurrounding area that is displayed to cause the trailer to be displayedwhen the cabin orientation and the trailer orientation are different.18. The display system according to claim 1, wherein the display screenand the surface are arranged such that part of the image displayed onthe display screen is reflected on the surface, and an image areareflected by the surface moves according to movement of a viewingposition of the viewer with respect to the surface.
 19. The displaysystem according to claim 1, wherein the display screen and the surfaceare arranged or optically designed to cause an optical distance from aposition of the viewer's eyes to a display face of the display screenvia the surface to be at least 1.1 meters or greater.
 20. The displaysystem according to claim 1, wherein the display screen is configuredsuch that a display face or illuminating light of the display face ofthe display screen is not visible to the viewer.
 21. The display systemaccording to claim 1, wherein a shape of the surface in a movementdirection of the portion of the surrounding area that is displayed is acurved shape projecting out towards the viewer.
 22. The display systemaccording to claim 21, wherein the surface is shaped with a smallercurve in a central region than an end region a movement direction of theviewer.
 23. The display system according to claim 1, further comprising:a control circuit configured to determine motion of the viewer andchange an orientation of the surface based on the motion of the viewerdetermined, the portion of the surrounding area that is displayedchanging in accordance with the change in orientation of the surface.24. The display system according to claim 1, wherein the surface is amirror.
 25. The display system according to claim 24, wherein the mirroris a semi-transmissive mirror.
 26. The display system according to claim25, wherein the semi-transmissive mirror includes a semi-transmissivelight dimming function, or a light dimming device that adjusts an amountof transmitted light, the light dimming device being disposed behind thesemi-transmissive mirror.
 27. The display system according to claim 1,further comprising at least two or more image capture devices attachedto an exterior of the vehicle and facing rearward with respect to atravel direction of the vehicle, wherein the image corresponding to asurrounding area of the vehicle is captured by at least one of the imagecapture devices.
 28. The display system according to claim 1, whereinthe vehicle is a car or a bus.
 29. A display control method comprising:displaying, with a display screen, an image corresponding to asurrounding area of a vehicle; and reflecting, with a surface, at leasta portion of the image displayed by the display screen to a viewer.